2023-09-30T00:47:03.570 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T00:47:03.570 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406800 --owner scheduled_yuriw@teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --verbose --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406800/orig.config.yaml 2023-09-30T00:47:03.602 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:47:03.833 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406800 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} email: yweinste@redhat.com first_in_suite: false job_id: '7406800' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T00:47:03.833 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T00:47:03.834 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T00:47:03.834 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T00:47:03.836 INFO:teuthology.task.internal:Checking packages... 2023-09-30T00:47:03.862 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T00:47:03.863 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T00:47:03.863 INFO:teuthology.packaging:ref: None 2023-09-30T00:47:03.863 INFO:teuthology.packaging:tag: None 2023-09-30T00:47:03.863 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:47:03.863 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:47:03.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:47:04.092 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:47:04.093 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T00:47:04.099 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T00:47:04.100 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T00:47:04.115 INFO:teuthology.task.internal:Saving configuration 2023-09-30T00:47:04.124 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T00:47:04.130 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:47:04.151 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406800', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:41:07.886654', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-09-30T00:47:04.151 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T00:47:04.158 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-09-30T00:47:04.158 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T00:47:04.197 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0f80c26ee0>, signals=[15]) 2023-09-30T00:47:04.197 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T00:47:04.204 INFO:teuthology.task.internal:Opening connections... 2023-09-30T00:47:04.204 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2023-09-30T00:47:04.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:47:04.256 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T00:47:04.262 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2023-09-30T00:47:04.376 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2023-09-30T00:47:04.377 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2023-09-30T00:47:04.382 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="Ubuntu" 2023-09-30T00:47:04.382 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-09-30T00:47:04.383 INFO:teuthology.orchestra.run.smithi186.stdout:ID=ubuntu 2023-09-30T00:47:04.383 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE=debian 2023-09-30T00:47:04.383 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-09-30T00:47:04.383 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="20.04" 2023-09-30T00:47:04.383 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-30T00:47:04.383 INFO:teuthology.orchestra.run.smithi186.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-30T00:47:04.384 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-30T00:47:04.384 INFO:teuthology.orchestra.run.smithi186.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-30T00:47:04.384 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_CODENAME=focal 2023-09-30T00:47:04.384 INFO:teuthology.orchestra.run.smithi186.stdout:UBUNTU_CODENAME=focal 2023-09-30T00:47:04.385 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2023-09-30T00:47:04.414 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T00:47:04.423 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T00:47:04.427 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T00:47:04.428 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:47:04.433 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T00:47:04.439 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T00:47:04.440 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:47:04.487 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T00:47:04.523 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T00:47:04.538 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T00:47:04.538 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T00:47:04.539 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:47:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:47:04.545 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2023-09-30T00:47:04.551 INFO:teuthology.orchestra.run.smithi186.stdout:5.4.0-124-generic 2023-09-30T00:47:04.551 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.4.0-124-generic 2023-09-30T00:47:04.552 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get clean 2023-09-30T00:47:04.667 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2023-09-30T00:47:04.976 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T00:47:04.977 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-09-30T00:47:05.034 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-09-30T00:47:05.377 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-09-30T00:47:05.446 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [652 kB] 2023-09-30T00:47:05.586 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [885 kB] 2023-09-30T00:47:05.789 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,464 kB] 2023-09-30T00:47:05.916 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,852 kB] 2023-09-30T00:47:05.975 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [385 kB] 2023-09-30T00:47:05.990 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-09-30T00:47:05.990 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-09-30T00:47:05.991 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,218 kB] 2023-09-30T00:47:06.058 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [309 kB] 2023-09-30T00:47:06.063 INFO:teuthology.orchestra.run.smithi186.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T00:47:06.064 INFO:teuthology.orchestra.run.smithi186.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [615 kB] 2023-09-30T00:47:06.075 INFO:teuthology.orchestra.run.smithi186.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [886 kB] 2023-09-30T00:47:06.090 INFO:teuthology.orchestra.run.smithi186.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [469 kB] 2023-09-30T00:47:06.103 INFO:teuthology.orchestra.run.smithi186.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-09-30T00:47:06.104 INFO:teuthology.orchestra.run.smithi186.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,330 kB] 2023-09-30T00:47:06.174 INFO:teuthology.orchestra.run.smithi186.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.7 kB] 2023-09-30T00:47:06.174 INFO:teuthology.orchestra.run.smithi186.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [325 kB] 2023-09-30T00:47:06.179 INFO:teuthology.orchestra.run.smithi186.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-09-30T00:47:06.274 INFO:teuthology.orchestra.run.smithi186.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [185 kB] 2023-09-30T00:47:06.344 INFO:teuthology.orchestra.run.smithi186.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,117 kB] 2023-09-30T00:47:06.690 INFO:teuthology.orchestra.run.smithi186.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-09-30T00:47:06.697 INFO:teuthology.orchestra.run.smithi186.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-09-30T00:47:06.704 INFO:teuthology.orchestra.run.smithi186.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-09-30T00:47:06.707 INFO:teuthology.orchestra.run.smithi186.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-09-30T00:47:06.709 INFO:teuthology.orchestra.run.smithi186.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-09-30T00:47:06.979 INFO:teuthology.orchestra.run.smithi186.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [748 kB] 2023-09-30T00:47:07.430 INFO:teuthology.orchestra.run.smithi186.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [267 kB] 2023-09-30T00:47:07.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-09-30T00:47:07.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-09-30T00:47:07.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-09-30T00:47:07.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-09-30T00:47:07.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-09-30T00:47:07.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-09-30T00:47:07.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-09-30T00:47:07.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-09-30T00:47:07.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-09-30T00:47:07.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-09-30T00:47:07.433 INFO:teuthology.orchestra.run.smithi186.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-09-30T00:47:09.719 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 17.4 MB in 3s (5,769 kB/s) 2023-09-30T00:47:11.074 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2023-09-30T00:47:11.100 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T00:47:11.172 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2023-09-30T00:47:11.372 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2023-09-30T00:47:11.373 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2023-09-30T00:47:11.550 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2023-09-30T00:47:11.551 INFO:teuthology.orchestra.run.smithi186.stdout: linux-generic linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T00:47:11.551 INFO:teuthology.orchestra.run.smithi186.stdout: linux-headers-generic linux-image-5.4.0-163-generic 2023-09-30T00:47:11.551 INFO:teuthology.orchestra.run.smithi186.stdout: linux-modules-5.4.0-163-generic linux-modules-extra-5.4.0-163-generic 2023-09-30T00:47:11.552 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2023-09-30T00:47:11.552 INFO:teuthology.orchestra.run.smithi186.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-09-30T00:47:11.552 INFO:teuthology.orchestra.run.smithi186.stdout:Recommended packages: 2023-09-30T00:47:11.552 INFO:teuthology.orchestra.run.smithi186.stdout: thermald 2023-09-30T00:47:11.599 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2023-09-30T00:47:11.599 INFO:teuthology.orchestra.run.smithi186.stdout: linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-30T00:47:11.599 INFO:teuthology.orchestra.run.smithi186.stdout: linux-image-5.4.0-163-generic linux-modules-5.4.0-163-generic 2023-09-30T00:47:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: linux-modules-extra-5.4.0-163-generic 2023-09-30T00:47:11.601 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages will be upgraded: 2023-09-30T00:47:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-30T00:47:11.658 INFO:teuthology.orchestra.run.smithi186.stdout:3 upgraded, 5 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T00:47:11.658 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 77.1 MB of archives. 2023-09-30T00:47:11.659 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 380 MB of additional disk space will be used. 2023-09-30T00:47:11.659 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-163-generic amd64 5.4.0-163.180 [15.0 MB] 2023-09-30T00:47:11.980 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-163-generic amd64 5.4.0-163.180 [10.5 MB] 2023-09-30T00:47:12.073 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-163-generic amd64 5.4.0-163.180 [39.2 MB] 2023-09-30T00:47:12.583 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.163.160 [1,904 B] 2023-09-30T00:47:12.584 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.163.160 [2,604 B] 2023-09-30T00:47:12.584 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163 all 5.4.0-163.180 [11.0 MB] 2023-09-30T00:47:13.054 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163-generic amd64 5.4.0-163.180 [1,358 kB] 2023-09-30T00:47:13.079 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.163.160 [2,468 B] 2023-09-30T00:47:13.756 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 77.1 MB in 1s (52.4 MB/s) 2023-09-30T00:47:13.929 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-modules-5.4.0-163-generic. 2023-09-30T00:47:15.628 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-09-30T00:47:15.633 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../0-linux-modules-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:47:15.718 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:47:17.752 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-image-5.4.0-163-generic. 2023-09-30T00:47:17.769 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../1-linux-image-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:47:17.865 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:47:18.930 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-163-generic. 2023-09-30T00:47:18.943 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:47:18.988 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:47:23.650 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../3-linux-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T00:47:23.776 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T00:47:24.109 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T00:47:24.229 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-image-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T00:47:24.599 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-headers-5.4.0-163. 2023-09-30T00:47:24.615 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../5-linux-headers-5.4.0-163_5.4.0-163.180_all.deb ... 2023-09-30T00:47:24.657 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T00:47:30.043 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-headers-5.4.0-163-generic. 2023-09-30T00:47:30.065 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../6-linux-headers-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-30T00:47:30.109 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:47:32.683 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.163.160_amd64.deb ... 2023-09-30T00:47:32.809 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-headers-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-30T00:47:33.162 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:47:34.378 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-30T00:47:34.499 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:47:35.651 INFO:teuthology.orchestra.run.smithi186.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-163-generic 2023-09-30T00:47:35.651 INFO:teuthology.orchestra.run.smithi186.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-163-generic 2023-09-30T00:47:35.784 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:47:35.911 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:47:37.095 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-image-generic (5.4.0.163.160) ... 2023-09-30T00:47:37.221 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-headers-generic (5.4.0.163.160) ... 2023-09-30T00:47:37.360 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-generic (5.4.0.163.160) ... 2023-09-30T00:47:37.482 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-30T00:47:37.554 INFO:teuthology.orchestra.run.smithi186.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-30T00:47:37.554 INFO:teuthology.orchestra.run.smithi186.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-163-generic 2023-09-30T00:48:00.398 INFO:teuthology.orchestra.run.smithi186.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-30T00:48:00.399 INFO:teuthology.orchestra.run.smithi186.stdout:Sourcing file `/etc/default/grub' 2023-09-30T00:48:00.412 INFO:teuthology.orchestra.run.smithi186.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T00:48:00.426 INFO:teuthology.orchestra.run.smithi186.stdout:Generating grub configuration file ... 2023-09-30T00:48:00.762 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T00:48:00.790 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T00:48:01.167 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T00:48:01.178 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T00:48:03.934 INFO:teuthology.orchestra.run.smithi186.stdout:done 2023-09-30T00:48:04.357 DEBUG:teuthology.orchestra.run.smithi186:> dpkg -s linux-image-generic 2023-09-30T00:48:04.407 INFO:teuthology.orchestra.run.smithi186.stdout:Package: linux-image-generic 2023-09-30T00:48:04.407 INFO:teuthology.orchestra.run.smithi186.stdout:Status: install ok installed 2023-09-30T00:48:04.407 INFO:teuthology.orchestra.run.smithi186.stdout:Priority: optional 2023-09-30T00:48:04.407 INFO:teuthology.orchestra.run.smithi186.stdout:Section: kernel 2023-09-30T00:48:04.407 INFO:teuthology.orchestra.run.smithi186.stdout:Installed-Size: 20 2023-09-30T00:48:04.407 INFO:teuthology.orchestra.run.smithi186.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T00:48:04.407 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: amd64 2023-09-30T00:48:04.407 INFO:teuthology.orchestra.run.smithi186.stdout:Source: linux-meta 2023-09-30T00:48:04.407 INFO:teuthology.orchestra.run.smithi186.stdout:Version: 5.4.0.163.160 2023-09-30T00:48:04.407 INFO:teuthology.orchestra.run.smithi186.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T00:48:04.408 INFO:teuthology.orchestra.run.smithi186.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T00:48:04.408 INFO:teuthology.orchestra.run.smithi186.stdout:Recommends: thermald 2023-09-30T00:48:04.408 INFO:teuthology.orchestra.run.smithi186.stdout:Description: Generic Linux kernel image 2023-09-30T00:48:04.408 INFO:teuthology.orchestra.run.smithi186.stdout: This package will always depend on the latest generic kernel image 2023-09-30T00:48:04.408 INFO:teuthology.orchestra.run.smithi186.stdout: available. 2023-09-30T00:48:04.409 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-163-generic 2023-09-30T00:48:04.409 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:48:04.409 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:48:04.409 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T00:48:04.409 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:48:04.409 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get clean 2023-09-30T00:48:04.508 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2023-09-30T00:48:04.787 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T00:48:04.787 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T00:48:04.857 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T00:48:04.947 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T00:48:06.532 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2023-09-30T00:48:06.558 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-30T00:48:06.625 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2023-09-30T00:48:06.820 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2023-09-30T00:48:06.821 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2023-09-30T00:48:07.023 INFO:teuthology.orchestra.run.smithi186.stdout:linux-image-generic is already the newest version (5.4.0.163.160). 2023-09-30T00:48:07.024 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 0 newly installed, 0 to remove and 263 not upgraded. 2023-09-30T00:48:07.025 DEBUG:teuthology.orchestra.run.smithi186:> dpkg -s linux-image-generic 2023-09-30T00:48:07.041 INFO:teuthology.orchestra.run.smithi186.stdout:Package: linux-image-generic 2023-09-30T00:48:07.042 INFO:teuthology.orchestra.run.smithi186.stdout:Status: install ok installed 2023-09-30T00:48:07.042 INFO:teuthology.orchestra.run.smithi186.stdout:Priority: optional 2023-09-30T00:48:07.042 INFO:teuthology.orchestra.run.smithi186.stdout:Section: kernel 2023-09-30T00:48:07.042 INFO:teuthology.orchestra.run.smithi186.stdout:Installed-Size: 20 2023-09-30T00:48:07.042 INFO:teuthology.orchestra.run.smithi186.stdout:Maintainer: Ubuntu Kernel Team 2023-09-30T00:48:07.042 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: amd64 2023-09-30T00:48:07.042 INFO:teuthology.orchestra.run.smithi186.stdout:Source: linux-meta 2023-09-30T00:48:07.043 INFO:teuthology.orchestra.run.smithi186.stdout:Version: 5.4.0.163.160 2023-09-30T00:48:07.043 INFO:teuthology.orchestra.run.smithi186.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-30T00:48:07.043 INFO:teuthology.orchestra.run.smithi186.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-30T00:48:07.043 INFO:teuthology.orchestra.run.smithi186.stdout:Recommends: thermald 2023-09-30T00:48:07.043 INFO:teuthology.orchestra.run.smithi186.stdout:Description: Generic Linux kernel image 2023-09-30T00:48:07.043 INFO:teuthology.orchestra.run.smithi186.stdout: This package will always depend on the latest generic kernel image 2023-09-30T00:48:07.043 INFO:teuthology.orchestra.run.smithi186.stdout: available. 2023-09-30T00:48:07.044 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2023-09-30T00:48:07.048 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.cijFI76ko6 2023-09-30T00:48:07.048 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub/grub.cfg /tmp/tmp.cijFI76ko6 2023-09-30T00:48:07.106 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.cijFI76ko6 2023-09-30T00:48:07.152 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.cijFI76ko6 is 9KB 2023-09-30T00:48:07.167 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.cijFI76ko6 2023-09-30T00:48:07.173 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-30T00:48:07.229 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-09-30T00:48:07.229 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-30T00:48:07.229 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-30T00:48:07.255 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-163-generic 2023-09-30T00:48:07.255 DEBUG:teuthology.orchestra.run.smithi186:> sudo update-grub 2023-09-30T00:48:08.550 INFO:teuthology.orchestra.run.smithi186.stderr:Sourcing file `/etc/default/grub' 2023-09-30T00:48:08.561 INFO:teuthology.orchestra.run.smithi186.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-30T00:48:08.573 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2023-09-30T00:48:08.823 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-30T00:48:08.835 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-30T00:48:09.187 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-30T00:48:09.199 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-30T00:48:10.835 INFO:teuthology.orchestra.run.smithi186.stderr:done 2023-09-30T00:48:10.837 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2023-09-30T00:48:40.866 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T00:48:40.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-09-30T00:48:40.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:48:59.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2023-09-30T00:49:08.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-09-30T00:49:08.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:49:11.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2023-09-30T00:49:23.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-09-30T00:49:23.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:49:41.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2023-09-30T00:49:56.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-09-30T00:49:56.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:49:56.976 ERROR:teuthology.orchestra.connection:Error authenticating with smithi186.front.sepia.ceph.com: No SSH private key found! 2023-09-30T00:49:56.976 DEBUG:teuthology.orchestra.remote:No existing session 2023-09-30T00:50:14.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-09-30T00:50:14.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:50:15.041 DEBUG:teuthology.orchestra.run.smithi186:> true 2023-09-30T00:50:15.653 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-09-30T00:50:15.653 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-163-generic"... 2023-09-30T00:50:15.654 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2023-09-30T00:50:15.660 INFO:teuthology.orchestra.run.smithi186.stdout:5.4.0-163-generic 2023-09-30T00:50:15.660 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-163-generic vs 5.4.0-163-generic 2023-09-30T00:50:15.660 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:50:15.660 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-30T00:50:16.662 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T00:50:16.662 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:50:16.779 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2023-09-30T00:50:16.804 DEBUG:teuthology.parallel:result is None 2023-09-30T00:50:16.804 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T00:50:16.811 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T00:50:16.811 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:50:16.818 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T00:50:16.838 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T00:50:16.846 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T00:50:16.846 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:50:16.867 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T00:50:16.874 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T00:50:16.874 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:50:16.923 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:50:16.933 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:50:16.935 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T00:50:16.941 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T00:50:16.941 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:50:16.953 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T00:50:16.962 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T00:50:16.962 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:50:16.969 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:50:17.016 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:50:17.023 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-09-30T00:50:17.023 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:50:17.045 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2023-09-30T00:50:17.095 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T00:50:17.102 INFO:teuthology.task.internal:Starting timer... 2023-09-30T00:50:17.102 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T00:50:17.126 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T00:50:17.136 DEBUG:teuthology.task.selinux:Excluding smithi186: OS 'ubuntu' does not support SELinux 2023-09-30T00:50:17.137 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T00:50:17.137 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-09-30T00:50:17.137 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T00:50:17.137 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T00:50:17.149 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-30T00:50:17.149 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T00:50:17.169 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T00:50:17.171 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T00:55:53.798 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2023-09-30T00:55:53.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-09-30T00:55:53.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:55:53.849 DEBUG:teuthology.orchestra.run.smithi186:> true 2023-09-30T00:55:53.972 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-09-30T00:55:53.973 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T00:55:53.988 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T00:55:53.988 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:55:53.989 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:55:54.024 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-09-30T00:55:54.024 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: Command line: ntpd -gq 2023-09-30T00:55:54.025 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: proto: precision = 0.154 usec (-23) 2023-09-30T00:55:54.025 INFO:teuthology.orchestra.run.smithi186.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T00:55:54.026 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-30T00:55:54.026 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: restrict ::: KOD does nothing without LIMITED. 2023-09-30T00:55:54.026 INFO:teuthology.orchestra.run.smithi186.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-30T00:55:54.027 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: Listen and drop on 0 v6wildcard [::]:123 2023-09-30T00:55:54.027 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-30T00:55:54.027 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: Listen normally on 2 lo 127.0.0.1:123 2023-09-30T00:55:54.027 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: Listen normally on 3 enp3s0f1 172.21.15.186:123 2023-09-30T00:55:54.027 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: Listen normally on 4 lo [::1]:123 2023-09-30T00:55:54.027 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6df5%5]:123 2023-09-30T00:55:54.028 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:55:54 ntpd[22755]: Listening on routing socket on fd #22 for interface updates 2023-09-30T00:55:55.024 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:55 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:55.024 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:55 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:55.025 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:55 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:55.025 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:55 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:55.025 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:55 ntpd[22755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:55:55.025 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:55 ntpd[22755]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:55:57.025 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:57.025 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:57.025 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:57.025 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:57.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:57.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:57.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:55:57.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:55:57.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:57.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:57.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:55:57.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:57 ntpd[22755]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:55:58.024 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:58 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:58.024 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:58 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:59.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:59.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:59.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:59.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:59.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:59.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:59.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:59.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:55:59.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:55:59 ntpd[22755]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:56:01.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:56:01.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:56:01.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:56:01.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:56:01.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:56:01.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:56:01.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:56:01.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:56:01.026 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:56:01.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:56:01.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:56:01.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:56:01.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:56:01.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:56:01.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-30T00:56:01.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: can't open /var/log/ntpstats/rawstats.20230930: Permission denied 2023-09-30T00:56:01.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-30T00:56:01.027 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: can't open /var/log/ntpstats/peerstats.20230930: Permission denied 2023-09-30T00:56:01.028 INFO:teuthology.orchestra.run.smithi186.stdout:30 Sep 00:56:01 ntpd[22755]: ntpd: time slew +0.000737 s 2023-09-30T00:56:01.028 INFO:teuthology.orchestra.run.smithi186.stdout:ntpd: time slew +0.000737s 2023-09-30T00:56:01.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-30T00:56:01.028 INFO:teuthology.orchestra.run.smithi186.stderr:30 Sep 00:56:01 ntpd[22755]: can't open /var/log/ntpstats/loopstats.20230930: Permission denied 2023-09-30T00:56:01.099 INFO:teuthology.orchestra.run.smithi186.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T00:56:01.099 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================== 2023-09-30T00:56:01.099 INFO:teuthology.orchestra.run.smithi186.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:56:01.099 INFO:teuthology.orchestra.run.smithi186.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:56:01.099 INFO:teuthology.orchestra.run.smithi186.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:56:01.100 INFO:teuthology.orchestra.run.smithi186.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-30T00:56:01.100 INFO:teuthology.run_tasks:Running task install... 2023-09-30T00:56:01.110 DEBUG:teuthology.task.install:project ceph 2023-09-30T00:56:01.111 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T00:56:01.111 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T00:56:01.111 INFO:teuthology.task.install:Using flavor: default 2023-09-30T00:56:01.118 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T00:56:01.118 INFO:teuthology.task.install:extra packages: [] 2023-09-30T00:56:01.118 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-key list | grep Ceph 2023-09-30T00:56:01.218 INFO:teuthology.orchestra.run.smithi186.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-09-30T00:56:01.275 INFO:teuthology.orchestra.run.smithi186.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-30T00:56:01.275 INFO:teuthology.orchestra.run.smithi186.stdout:uid [ unknown] Ceph.com (release key) 2023-09-30T00:56:01.276 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-30T00:56:01.277 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:56:01.496 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default/ 2023-09-30T00:56:01.496 INFO:teuthology.task.install.deb:Package version is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:56:01.666 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-09-30T00:56:01.666 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-30T00:56:01.683 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2023-09-30T00:56:01.936 INFO:teuthology.orchestra.run.smithi186.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal InRelease 2023-09-30T00:56:01.951 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-30T00:56:01.951 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-30T00:56:01.972 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-09-30T00:56:02.010 INFO:teuthology.orchestra.run.smithi186.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal Release.gpg 2023-09-30T00:56:02.021 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-30T00:56:02.047 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main i386 Packages [3,375 B] 2023-09-30T00:56:02.103 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-30T00:56:02.105 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-09-30T00:56:02.608 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 28.4 kB in 1s (37.3 kB/s) 2023-09-30T00:56:03.639 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2023-09-30T00:56:03.663 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1165-g93dce5e1-1focal cephadm=17.2.6-1165-g93dce5e1-1focal ceph-mds=17.2.6-1165-g93dce5e1-1focal ceph-mgr=17.2.6-1165-g93dce5e1-1focal ceph-common=17.2.6-1165-g93dce5e1-1focal ceph-fuse=17.2.6-1165-g93dce5e1-1focal ceph-test=17.2.6-1165-g93dce5e1-1focal ceph-volume=17.2.6-1165-g93dce5e1-1focal radosgw=17.2.6-1165-g93dce5e1-1focal python3-rados=17.2.6-1165-g93dce5e1-1focal python3-rgw=17.2.6-1165-g93dce5e1-1focal python3-cephfs=17.2.6-1165-g93dce5e1-1focal python3-rbd=17.2.6-1165-g93dce5e1-1focal libcephfs2=17.2.6-1165-g93dce5e1-1focal libcephfs-dev=17.2.6-1165-g93dce5e1-1focal librados2=17.2.6-1165-g93dce5e1-1focal librbd1=17.2.6-1165-g93dce5e1-1focal rbd-fuse=17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:56:03.731 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2023-09-30T00:56:03.931 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2023-09-30T00:56:03.931 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2023-09-30T00:56:04.111 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages were automatically installed and are no longer required: 2023-09-30T00:56:04.112 INFO:teuthology.orchestra.run.smithi186.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-30T00:56:04.112 INFO:teuthology.orchestra.run.smithi186.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-30T00:56:04.112 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2023-09-30T00:56:04.112 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-30T00:56:04.112 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-09-30T00:56:04.112 INFO:teuthology.orchestra.run.smithi186.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-09-30T00:56:04.114 INFO:teuthology.orchestra.run.smithi186.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-09-30T00:56:04.114 INFO:teuthology.orchestra.run.smithi186.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-09-30T00:56:04.114 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-09-30T00:56:04.115 INFO:teuthology.orchestra.run.smithi186.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-09-30T00:56:04.115 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-09-30T00:56:04.115 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-09-30T00:56:04.115 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-09-30T00:56:04.115 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-09-30T00:56:04.115 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-09-30T00:56:04.115 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-09-30T00:56:04.115 INFO:teuthology.orchestra.run.smithi186.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-09-30T00:56:04.115 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-09-30T00:56:04.115 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-09-30T00:56:04.115 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout: socat xmlstarlet 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-09-30T00:56:04.116 INFO:teuthology.orchestra.run.smithi186.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-09-30T00:56:04.117 INFO:teuthology.orchestra.run.smithi186.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-09-30T00:56:04.117 INFO:teuthology.orchestra.run.smithi186.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-09-30T00:56:04.117 INFO:teuthology.orchestra.run.smithi186.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-09-30T00:56:04.117 INFO:teuthology.orchestra.run.smithi186.stdout:Recommended packages: 2023-09-30T00:56:04.117 INFO:teuthology.orchestra.run.smithi186.stdout: btrfs-tools 2023-09-30T00:56:04.231 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2023-09-30T00:56:04.232 INFO:teuthology.orchestra.run.smithi186.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-30T00:56:04.232 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-30T00:56:04.232 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-09-30T00:56:04.234 INFO:teuthology.orchestra.run.smithi186.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-09-30T00:56:04.234 INFO:teuthology.orchestra.run.smithi186.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-09-30T00:56:04.234 INFO:teuthology.orchestra.run.smithi186.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-09-30T00:56:04.234 INFO:teuthology.orchestra.run.smithi186.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-09-30T00:56:04.234 INFO:teuthology.orchestra.run.smithi186.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-09-30T00:56:04.234 INFO:teuthology.orchestra.run.smithi186.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-09-30T00:56:04.234 INFO:teuthology.orchestra.run.smithi186.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-09-30T00:56:04.235 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-09-30T00:56:04.236 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-09-30T00:56:04.236 INFO:teuthology.orchestra.run.smithi186.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-09-30T00:56:04.236 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages will be upgraded: 2023-09-30T00:56:04.236 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 librbd1 2023-09-30T00:56:04.303 INFO:teuthology.orchestra.run.smithi186.stdout:2 upgraded, 108 newly installed, 0 to remove and 245 not upgraded. 2023-09-30T00:56:04.303 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 209 MB of archives. 2023-09-30T00:56:04.303 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 980 MB of additional disk space will be used. 2023-09-30T00:56:04.303 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-09-30T00:56:04.399 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-09-30T00:56:04.401 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-09-30T00:56:04.411 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-09-30T00:56:04.419 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-09-30T00:56:04.446 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-09-30T00:56:04.447 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-09-30T00:56:04.454 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-09-30T00:56:04.456 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1165-g93dce5e1-1focal [3,501 kB] 2023-09-30T00:56:04.490 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-09-30T00:56:04.492 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-09-30T00:56:04.498 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-09-30T00:56:04.500 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-09-30T00:56:04.500 INFO:teuthology.orchestra.run.smithi186.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-09-30T00:56:04.503 INFO:teuthology.orchestra.run.smithi186.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-09-30T00:56:04.503 INFO:teuthology.orchestra.run.smithi186.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-09-30T00:56:04.503 INFO:teuthology.orchestra.run.smithi186.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-09-30T00:56:04.508 INFO:teuthology.orchestra.run.smithi186.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-09-30T00:56:04.509 INFO:teuthology.orchestra.run.smithi186.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-09-30T00:56:04.528 INFO:teuthology.orchestra.run.smithi186.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-09-30T00:56:04.529 INFO:teuthology.orchestra.run.smithi186.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-09-30T00:56:04.530 INFO:teuthology.orchestra.run.smithi186.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-09-30T00:56:04.532 INFO:teuthology.orchestra.run.smithi186.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-30T00:56:04.532 INFO:teuthology.orchestra.run.smithi186.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-09-30T00:56:04.532 INFO:teuthology.orchestra.run.smithi186.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-09-30T00:56:04.536 INFO:teuthology.orchestra.run.smithi186.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-09-30T00:56:04.536 INFO:teuthology.orchestra.run.smithi186.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-09-30T00:56:04.536 INFO:teuthology.orchestra.run.smithi186.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-09-30T00:56:04.551 INFO:teuthology.orchestra.run.smithi186.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-09-30T00:56:04.552 INFO:teuthology.orchestra.run.smithi186.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-09-30T00:56:04.558 INFO:teuthology.orchestra.run.smithi186.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-09-30T00:56:04.571 INFO:teuthology.orchestra.run.smithi186.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-09-30T00:56:04.571 INFO:teuthology.orchestra.run.smithi186.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-09-30T00:56:04.572 INFO:teuthology.orchestra.run.smithi186.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-09-30T00:56:04.575 INFO:teuthology.orchestra.run.smithi186.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-09-30T00:56:04.577 INFO:teuthology.orchestra.run.smithi186.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-09-30T00:56:04.579 INFO:teuthology.orchestra.run.smithi186.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-30T00:56:04.579 INFO:teuthology.orchestra.run.smithi186.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-09-30T00:56:04.590 INFO:teuthology.orchestra.run.smithi186.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-09-30T00:56:04.591 INFO:teuthology.orchestra.run.smithi186.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-09-30T00:56:04.592 INFO:teuthology.orchestra.run.smithi186.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-09-30T00:56:04.610 INFO:teuthology.orchestra.run.smithi186.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-09-30T00:56:04.611 INFO:teuthology.orchestra.run.smithi186.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-09-30T00:56:04.715 INFO:teuthology.orchestra.run.smithi186.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-09-30T00:56:04.786 INFO:teuthology.orchestra.run.smithi186.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-09-30T00:56:04.788 INFO:teuthology.orchestra.run.smithi186.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-09-30T00:56:04.805 INFO:teuthology.orchestra.run.smithi186.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-09-30T00:56:04.805 INFO:teuthology.orchestra.run.smithi186.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-09-30T00:56:04.805 INFO:teuthology.orchestra.run.smithi186.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-09-30T00:56:04.806 INFO:teuthology.orchestra.run.smithi186.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-09-30T00:56:04.806 INFO:teuthology.orchestra.run.smithi186.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-09-30T00:56:04.806 INFO:teuthology.orchestra.run.smithi186.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-09-30T00:56:04.807 INFO:teuthology.orchestra.run.smithi186.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-09-30T00:56:04.808 INFO:teuthology.orchestra.run.smithi186.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-09-30T00:56:04.826 INFO:teuthology.orchestra.run.smithi186.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-09-30T00:56:04.826 INFO:teuthology.orchestra.run.smithi186.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-09-30T00:56:04.829 INFO:teuthology.orchestra.run.smithi186.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-09-30T00:56:04.845 INFO:teuthology.orchestra.run.smithi186.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-09-30T00:56:04.847 INFO:teuthology.orchestra.run.smithi186.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-09-30T00:56:04.848 INFO:teuthology.orchestra.run.smithi186.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-09-30T00:56:04.852 INFO:teuthology.orchestra.run.smithi186.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-09-30T00:56:04.855 INFO:teuthology.orchestra.run.smithi186.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-09-30T00:56:04.872 INFO:teuthology.orchestra.run.smithi186.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-09-30T00:56:04.875 INFO:teuthology.orchestra.run.smithi186.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-09-30T00:56:04.929 INFO:teuthology.orchestra.run.smithi186.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-09-30T00:56:04.930 INFO:teuthology.orchestra.run.smithi186.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-09-30T00:56:04.930 INFO:teuthology.orchestra.run.smithi186.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-09-30T00:56:04.931 INFO:teuthology.orchestra.run.smithi186.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-09-30T00:56:04.931 INFO:teuthology.orchestra.run.smithi186.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-09-30T00:56:04.932 INFO:teuthology.orchestra.run.smithi186.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-09-30T00:56:04.946 INFO:teuthology.orchestra.run.smithi186.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-09-30T00:56:04.946 INFO:teuthology.orchestra.run.smithi186.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-30T00:56:04.989 INFO:teuthology.orchestra.run.smithi186.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1165-g93dce5e1-1focal [3,552 kB] 2023-09-30T00:56:05.013 INFO:teuthology.orchestra.run.smithi186.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-09-30T00:56:05.013 INFO:teuthology.orchestra.run.smithi186.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-09-30T00:56:05.014 INFO:teuthology.orchestra.run.smithi186.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-09-30T00:56:05.015 INFO:teuthology.orchestra.run.smithi186.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-09-30T00:56:05.015 INFO:teuthology.orchestra.run.smithi186.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-09-30T00:56:05.016 INFO:teuthology.orchestra.run.smithi186.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-09-30T00:56:05.018 INFO:teuthology.orchestra.run.smithi186.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-09-30T00:56:05.019 INFO:teuthology.orchestra.run.smithi186.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-09-30T00:56:05.244 INFO:teuthology.orchestra.run.smithi186.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1165-g93dce5e1-1focal [758 kB] 2023-09-30T00:56:05.283 INFO:teuthology.orchestra.run.smithi186.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1165-g93dce5e1-1focal [414 kB] 2023-09-30T00:56:05.315 INFO:teuthology.orchestra.run.smithi186.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1165-g93dce5e1-1focal [22.5 kB] 2023-09-30T00:56:05.316 INFO:teuthology.orchestra.run.smithi186.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1165-g93dce5e1-1focal [200 kB] 2023-09-30T00:56:05.321 INFO:teuthology.orchestra.run.smithi186.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1165-g93dce5e1-1focal [47.1 kB] 2023-09-30T00:56:05.322 INFO:teuthology.orchestra.run.smithi186.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1165-g93dce5e1-1focal [385 kB] 2023-09-30T00:56:05.351 INFO:teuthology.orchestra.run.smithi186.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1165-g93dce5e1-1focal [5,569 kB] 2023-09-30T00:56:05.606 INFO:teuthology.orchestra.run.smithi186.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1165-g93dce5e1-1focal [116 kB] 2023-09-30T00:56:05.608 INFO:teuthology.orchestra.run.smithi186.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1165-g93dce5e1-1focal [480 kB] 2023-09-30T00:56:05.616 INFO:teuthology.orchestra.run.smithi186.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1165-g93dce5e1-1focal [27.7 MB] 2023-09-30T00:56:06.613 INFO:teuthology.orchestra.run.smithi186.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1165-g93dce5e1-1focal [6,453 kB] 2023-09-30T00:56:06.749 INFO:teuthology.orchestra.run.smithi186.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1165-g93dce5e1-1focal [221 kB] 2023-09-30T00:56:06.755 INFO:teuthology.orchestra.run.smithi186.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1165-g93dce5e1-1focal [153 kB] 2023-09-30T00:56:06.760 INFO:teuthology.orchestra.run.smithi186.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1165-g93dce5e1-1focal [1,589 kB] 2023-09-30T00:56:06.812 INFO:teuthology.orchestra.run.smithi186.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1165-g93dce5e1-1focal [7,845 kB] 2023-09-30T00:56:07.112 INFO:teuthology.orchestra.run.smithi186.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1165-g93dce5e1-1focal [28.7 MB] 2023-09-30T00:56:08.128 INFO:teuthology.orchestra.run.smithi186.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1165-g93dce5e1-1focal [3,928 B] 2023-09-30T00:56:08.128 INFO:teuthology.orchestra.run.smithi186.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1165-g93dce5e1-1focal [868 kB] 2023-09-30T00:56:08.141 INFO:teuthology.orchestra.run.smithi186.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1165-g93dce5e1-1focal [2,404 kB] 2023-09-30T00:56:08.158 INFO:teuthology.orchestra.run.smithi186.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1165-g93dce5e1-1focal [78.9 kB] 2023-09-30T00:56:08.159 INFO:teuthology.orchestra.run.smithi186.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1165-g93dce5e1-1focal [117 kB] 2023-09-30T00:56:08.160 INFO:teuthology.orchestra.run.smithi186.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1165-g93dce5e1-1focal [1,760 kB] 2023-09-30T00:56:08.215 INFO:teuthology.orchestra.run.smithi186.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1165-g93dce5e1-1focal [7,861 kB] 2023-09-30T00:56:08.474 INFO:teuthology.orchestra.run.smithi186.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1165-g93dce5e1-1focal [4,048 B] 2023-09-30T00:56:08.474 INFO:teuthology.orchestra.run.smithi186.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1165-g93dce5e1-1focal [56.2 MB] 2023-09-30T00:56:10.634 INFO:teuthology.orchestra.run.smithi186.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1165-g93dce5e1-1focal [229 kB] 2023-09-30T00:56:10.638 INFO:teuthology.orchestra.run.smithi186.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1165-g93dce5e1-1focal [21.9 kB] 2023-09-30T00:56:10.639 INFO:teuthology.orchestra.run.smithi186.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1165-g93dce5e1-1focal [17.0 MB] 2023-09-30T00:56:11.174 INFO:teuthology.orchestra.run.smithi186.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1165-g93dce5e1-1focal [79.5 kB] 2023-09-30T00:56:11.799 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 209 MB in 7s (30.2 MB/s) 2023-09-30T00:56:11.993 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-09-30T00:56:12.049 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141758 files and directories currently installed.) 2023-09-30T00:56:12.053 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-09-30T00:56:12.142 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T00:56:12.503 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../001-librbd1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:12.629 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librbd1 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T00:56:13.344 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../002-librados2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:13.467 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librados2 (17.2.6-1165-g93dce5e1-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-30T00:56:14.162 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libcephfs2. 2023-09-30T00:56:14.178 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../003-libcephfs2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:14.220 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:14.565 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rados. 2023-09-30T00:56:14.582 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../004-python3-rados_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:14.614 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:14.967 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-30T00:56:14.983 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:56:15.025 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:15.294 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cephfs. 2023-09-30T00:56:15.311 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../006-python3-cephfs_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:15.352 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:15.714 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-30T00:56:15.733 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:56:15.772 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:16.092 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-prettytable. 2023-09-30T00:56:16.108 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-09-30T00:56:16.150 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-09-30T00:56:16.436 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rbd. 2023-09-30T00:56:16.452 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../009-python3-rbd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:16.494 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:16.843 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-09-30T00:56:16.861 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-09-30T00:56:16.902 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T00:56:17.285 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-09-30T00:56:17.302 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-09-30T00:56:17.343 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T00:56:17.693 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-30T00:56:17.710 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-09-30T00:56:17.752 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T00:56:18.121 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-30T00:56:18.138 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-09-30T00:56:18.179 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T00:56:18.558 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-30T00:56:18.574 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-09-30T00:56:18.616 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T00:56:19.087 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-30T00:56:19.103 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T00:56:19.154 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:56:19.706 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-30T00:56:19.722 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T00:56:19.764 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:56:20.093 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-30T00:56:20.109 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-30T00:56:20.151 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:56:20.504 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-09-30T00:56:20.522 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-09-30T00:56:20.562 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T00:56:20.932 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librgw2. 2023-09-30T00:56:20.948 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../019-librgw2_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:20.990 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:21.759 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rgw. 2023-09-30T00:56:21.777 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../020-python3-rgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:21.817 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:22.145 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liboath0. 2023-09-30T00:56:22.162 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-09-30T00:56:22.199 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-09-30T00:56:22.665 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libradosstriper1. 2023-09-30T00:56:22.682 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../022-libradosstriper1_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:22.723 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:23.168 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-common. 2023-09-30T00:56:23.188 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../023-ceph-common_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:23.226 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:25.916 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-base. 2023-09-30T00:56:25.936 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../024-ceph-base_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:26.010 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:27.004 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cryptography. 2023-09-30T00:56:27.021 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-09-30T00:56:27.062 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T00:56:27.490 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-openssl. 2023-09-30T00:56:27.507 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-09-30T00:56:27.548 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-09-30T00:56:27.784 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-30T00:56:27.802 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:56:27.835 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:28.272 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-30T00:56:28.292 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-09-30T00:56:28.330 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T00:56:28.658 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-30T00:56:28.677 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-09-30T00:56:28.717 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-09-30T00:56:29.037 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-natsort. 2023-09-30T00:56:29.057 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-09-30T00:56:29.095 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-09-30T00:56:29.449 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jwt. 2023-09-30T00:56:29.466 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-09-30T00:56:29.506 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T00:56:29.826 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-logutils. 2023-09-30T00:56:29.845 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-09-30T00:56:29.884 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-09-30T00:56:30.187 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-30T00:56:30.203 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-09-30T00:56:30.245 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-09-30T00:56:30.590 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-mako. 2023-09-30T00:56:30.606 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-09-30T00:56:30.648 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T00:56:30.960 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-30T00:56:30.977 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-30T00:56:31.018 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-30T00:56:31.287 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-30T00:56:31.307 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-09-30T00:56:31.345 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T00:56:31.657 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webob. 2023-09-30T00:56:31.674 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-09-30T00:56:31.740 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-09-30T00:56:32.068 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-soupsieve. 2023-09-30T00:56:32.087 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-09-30T00:56:32.126 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T00:56:32.454 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-bs4. 2023-09-30T00:56:32.470 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-09-30T00:56:32.512 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-09-30T00:56:32.849 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-waitress. 2023-09-30T00:56:32.866 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-09-30T00:56:32.902 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T00:56:33.227 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-tempita. 2023-09-30T00:56:33.243 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-09-30T00:56:33.277 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-09-30T00:56:33.579 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-paste. 2023-09-30T00:56:33.595 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-09-30T00:56:33.662 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T00:56:33.974 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-30T00:56:33.993 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-09-30T00:56:34.024 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T00:56:34.352 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-30T00:56:34.367 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-09-30T00:56:34.410 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-09-30T00:56:34.713 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webtest. 2023-09-30T00:56:34.730 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-09-30T00:56:34.771 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-09-30T00:56:35.124 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pecan. 2023-09-30T00:56:35.140 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-09-30T00:56:35.182 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T00:56:35.520 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-30T00:56:35.538 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-09-30T00:56:35.595 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T00:56:35.948 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-30T00:56:35.966 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:36.006 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:36.393 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr. 2023-09-30T00:56:36.410 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../049-ceph-mgr_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:36.451 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:36.955 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mon. 2023-09-30T00:56:36.971 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../050-ceph-mon_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:37.012 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:38.866 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-osd. 2023-09-30T00:56:38.885 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../051-ceph-osd_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:38.924 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:41.752 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph. 2023-09-30T00:56:41.772 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../052-ceph_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:41.810 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:42.154 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-fuse. 2023-09-30T00:56:42.170 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../053-ceph-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:42.212 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:42.674 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mds. 2023-09-30T00:56:42.691 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../054-ceph-mds_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:42.732 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:43.294 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package cephadm. 2023-09-30T00:56:43.313 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../055-cephadm_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:43.352 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:43.663 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jinja2. 2023-09-30T00:56:43.679 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-09-30T00:56:43.763 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-09-30T00:56:44.124 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-30T00:56:44.140 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:56:44.182 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:44.519 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-30T00:56:44.535 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-09-30T00:56:44.593 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-30T00:56:44.846 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjs-underscore. 2023-09-30T00:56:44.865 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-09-30T00:56:44.904 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T00:56:45.206 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-09-30T00:56:45.225 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-09-30T00:56:45.273 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T00:56:45.592 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-routes. 2023-09-30T00:56:45.610 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-09-30T00:56:45.651 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-09-30T00:56:46.004 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-30T00:56:46.019 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:56:46.062 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:46.582 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-decorator. 2023-09-30T00:56:46.598 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-09-30T00:56:46.640 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T00:56:46.960 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-09-30T00:56:46.977 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-09-30T00:56:47.018 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T00:56:47.354 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-scipy. 2023-09-30T00:56:47.370 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-09-30T00:56:47.412 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-09-30T00:56:48.698 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-09-30T00:56:48.714 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-09-30T00:56:48.756 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T00:56:49.318 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-joblib. 2023-09-30T00:56:49.337 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-09-30T00:56:49.367 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-09-30T00:56:49.746 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-sklearn. 2023-09-30T00:56:49.762 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-09-30T00:56:49.804 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T00:56:50.562 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-30T00:56:50.580 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:56:50.620 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:51.757 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cachetools. 2023-09-30T00:56:51.777 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-09-30T00:56:51.815 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-09-30T00:56:52.101 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-30T00:56:52.120 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-09-30T00:56:52.159 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T00:56:52.504 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-30T00:56:52.520 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-09-30T00:56:52.562 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T00:56:52.906 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rsa. 2023-09-30T00:56:52.923 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-09-30T00:56:52.964 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rsa (4.0-3) ... 2023-09-30T00:56:53.284 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-google-auth. 2023-09-30T00:56:53.302 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-09-30T00:56:53.343 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-09-30T00:56:53.704 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-blinker. 2023-09-30T00:56:53.720 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-09-30T00:56:53.762 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T00:56:54.415 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-oauthlib. 2023-09-30T00:56:54.431 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-09-30T00:56:54.473 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T00:56:54.843 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-30T00:56:54.861 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-09-30T00:56:54.902 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T00:56:55.213 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-websocket. 2023-09-30T00:56:55.230 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-09-30T00:56:55.271 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T00:56:55.599 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-30T00:56:55.615 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-09-30T00:56:55.657 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T00:56:56.237 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-30T00:56:56.254 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:56:56.295 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:56:56.718 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-30T00:56:56.737 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-09-30T00:56:56.775 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-09-30T00:56:57.128 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-30T00:56:57.148 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T00:56:57.186 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:56:57.481 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package jq. 2023-09-30T00:56:57.498 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-30T00:56:57.539 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:56:57.775 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package socat. 2023-09-30T00:56:57.793 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-09-30T00:56:57.824 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking socat (1.7.3.3-2) ... 2023-09-30T00:56:58.145 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package xmlstarlet. 2023-09-30T00:56:58.163 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-09-30T00:56:58.203 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-09-30T00:56:58.500 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-test. 2023-09-30T00:56:58.516 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../086-ceph-test_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:56:58.549 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:03.618 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-volume. 2023-09-30T00:57:03.637 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../087-ceph-volume_17.2.6-1165-g93dce5e1-1focal_all.deb ... 2023-09-30T00:57:03.676 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:04.006 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package fonts-mathjax. 2023-09-30T00:57:04.025 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T00:57:04.064 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:57:04.601 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-30T00:57:04.619 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:57:04.659 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:05.046 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package nvme-cli. 2023-09-30T00:57:05.063 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-09-30T00:57:05.132 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T00:57:05.442 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjs-mathjax. 2023-09-30T00:57:05.462 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-30T00:57:05.492 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:57:07.118 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python-natsort-doc. 2023-09-30T00:57:07.135 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-09-30T00:57:07.174 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-09-30T00:57:07.502 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-atomicwrites. 2023-09-30T00:57:07.520 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-09-30T00:57:07.560 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T00:57:07.897 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-attr. 2023-09-30T00:57:07.915 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-09-30T00:57:07.955 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-09-30T00:57:08.349 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webencodings. 2023-09-30T00:57:08.369 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-09-30T00:57:08.417 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T00:57:08.685 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-html5lib. 2023-09-30T00:57:08.702 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-09-30T00:57:08.735 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-09-30T00:57:09.088 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-09-30T00:57:09.105 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-09-30T00:57:09.146 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T00:57:09.575 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-packaging. 2023-09-30T00:57:09.592 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-09-30T00:57:09.649 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-packaging (20.3-1) ... 2023-09-30T00:57:09.952 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pastescript. 2023-09-30T00:57:09.969 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-09-30T00:57:10.010 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-30T00:57:10.364 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pluggy. 2023-09-30T00:57:10.380 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-09-30T00:57:10.413 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-09-30T00:57:10.716 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-psutil. 2023-09-30T00:57:10.735 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-09-30T00:57:10.774 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T00:57:11.111 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-py. 2023-09-30T00:57:11.130 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-09-30T00:57:11.169 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T00:57:11.481 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-30T00:57:11.499 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-09-30T00:57:11.573 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T00:57:11.884 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-30T00:57:11.905 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-09-30T00:57:11.942 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T00:57:12.262 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pytest. 2023-09-30T00:57:12.279 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-09-30T00:57:12.319 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-09-30T00:57:12.657 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-simplejson. 2023-09-30T00:57:12.673 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-09-30T00:57:12.714 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T00:57:13.026 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-30T00:57:13.046 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-09-30T00:57:13.084 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T00:57:13.841 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package radosgw. 2023-09-30T00:57:13.857 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../108-radosgw_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:57:13.898 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:15.652 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package rbd-fuse. 2023-09-30T00:57:15.671 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../109-rbd-fuse_17.2.6-1165-g93dce5e1-1focal_amd64.deb ... 2023-09-30T00:57:15.710 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:16.029 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-attr (19.3.0-2) ... 2023-09-30T00:57:16.318 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-09-30T00:57:16.428 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-30T00:57:16.555 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-09-30T00:57:16.780 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:17.450 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user cephadm....done 2023-09-30T00:57:17.587 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:57:17.713 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-30T00:57:18.098 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-30T00:57:18.400 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-30T00:57:18.509 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-09-30T00:57:18.655 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-09-30T00:57:18.982 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-09-30T00:57:19.401 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-30T00:57:19.677 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-09-30T00:57:19.955 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-30T00:57:20.247 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-30T00:57:20.555 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-ceph-argparse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:20.842 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-30T00:57:20.943 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-09-30T00:57:21.203 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-30T00:57:21.479 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-30T00:57:21.605 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-09-30T00:57:21.981 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-09-30T00:57:22.274 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-09-30T00:57:22.559 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-30T00:57:22.885 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-09-30T00:57:23.153 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-09-30T00:57:23.470 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-09-30T00:57:23.763 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-packaging (20.3-1) ... 2023-09-30T00:57:24.056 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-09-30T00:57:24.190 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-30T00:57:24.316 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-30T00:57:24.618 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-09-30T00:57:25.369 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-30T00:57:25.780 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-30T00:57:26.173 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up socat (1.7.3.3-2) ... 2023-09-30T00:57:26.299 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:26.617 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:57:26.727 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-09-30T00:57:27.063 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-30T00:57:27.223 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-09-30T00:57:27.350 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-09-30T00:57:27.626 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-30T00:57:27.902 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:57:28.003 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-30T00:57:28.121 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-09-30T00:57:28.451 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-09-30T00:57:28.693 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-09-30T00:57:28.953 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-30T00:57:29.087 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-30T00:57:29.363 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-09-30T00:57:29.648 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-30T00:57:29.982 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-09-30T00:57:30.388 INFO:teuthology.orchestra.run.smithi186.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-09-30T00:57:30.671 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-09-30T00:57:30.797 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-30T00:57:31.106 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-09-30T00:57:31.466 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-30T00:57:31.625 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-30T00:57:31.751 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-09-30T00:57:32.069 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-30T00:57:32.346 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librados2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:32.472 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-09-30T00:57:33.936 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:34.036 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:57:34.196 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-09-30T00:57:34.556 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-09-30T00:57:34.849 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-09-30T00:57:35.158 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-30T00:57:35.284 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-09-30T00:57:35.594 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libcephfs2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:35.695 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-30T00:57:35.997 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libradosstriper1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:36.148 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-30T00:57:36.474 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-30T00:57:37.504 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librbd1 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:37.630 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-30T00:57:37.756 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-09-30T00:57:37.883 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-09-30T00:57:38.034 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rsa (4.0-3) ... 2023-09-30T00:57:38.335 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-30T00:57:38.436 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-09-30T00:57:38.712 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-modules-core (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:38.821 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:39.046 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T00:57:39.046 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-30T00:57:39.514 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libcephfs-dev (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:39.776 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-30T00:57:40.070 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-30T00:57:40.730 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-09-30T00:57:41.099 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rados (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:41.226 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librgw2 (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:41.377 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rbd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:41.503 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up rbd-fuse (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:41.629 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-routes (2.4.1-2) ... 2023-09-30T00:57:41.947 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-09-30T00:57:42.265 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:42.374 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cephfs (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:42.483 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-09-30T00:57:43.420 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-common (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:43.940 INFO:teuthology.orchestra.run.smithi186.stdout:Adding group ceph....done 2023-09-30T00:57:44.361 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user ceph....done 2023-09-30T00:57:44.529 INFO:teuthology.orchestra.run.smithi186.stdout:Setting system user ceph properties....done 2023-09-30T00:57:44.547 INFO:teuthology.orchestra.run.smithi186.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-30T00:57:44.690 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-30T00:57:45.003 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-30T00:57:45.483 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-test (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:45.610 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up radosgw (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:45.913 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T00:57:45.913 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-30T00:57:46.561 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-base (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:46.910 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-30T00:57:47.371 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mds (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:47.598 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T00:57:47.598 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-30T00:57:48.048 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:48.273 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T00:57:48.273 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-30T00:57:48.741 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-osd (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:49.232 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T00:57:49.232 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-30T00:57:49.684 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:49.810 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:49.938 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mon (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:50.159 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T00:57:50.160 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-30T00:57:50.590 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-cephadm (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:50.734 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:50.884 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-dashboard (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:51.026 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-volume (17.2.6-1165-g93dce5e1-1focal) ... 2023-09-30T00:57:51.338 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-30T00:57:51.514 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-09-30T00:57:51.807 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-30T00:57:54.744 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-09-30T00:57:55.342 INFO:teuthology.orchestra.run.smithi186.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-30T00:57:55.344 DEBUG:teuthology.parallel:result is None 2023-09-30T00:57:55.345 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:57:55.572 DEBUG:teuthology.orchestra.run.smithi186:> dpkg-query -W -f '${Version}' ceph 2023-09-30T00:57:55.596 INFO:teuthology.orchestra.run.smithi186.stdout:17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:57:55.596 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165-g93dce5e1-1focal 2023-09-30T00:57:55.596 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165-g93dce5e1-1focal is installed. 2023-09-30T00:57:55.597 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T00:57:55.597 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-09-30T00:57:55.597 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:57:55.608 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T00:57:55.609 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-09-30T00:57:55.609 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:57:55.624 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:57:55.639 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T00:57:55.639 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-09-30T00:57:55.639 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:57:55.698 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:57:55.707 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T00:57:55.708 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-09-30T00:57:55.708 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:57:55.719 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:57:55.779 INFO:teuthology.run_tasks:Running task workunit... 2023-09-30T00:57:55.791 INFO:tasks.workunit:Pulling workunits from ref 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:57:55.792 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-09-30T00:57:55.792 INFO:tasks.workunit:timeout=3h 2023-09-30T00:57:55.792 INFO:tasks.workunit:cleanup=True 2023-09-30T00:57:55.792 DEBUG:teuthology.orchestra.run.smithi186:> stat -- /home/ubuntu/cephtest/mnt.0 2023-09-30T00:57:55.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:57:55.799 INFO:teuthology.orchestra.run.smithi186.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-09-30T00:57:55.800 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-09-30T00:57:55.849 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-09-30T00:57:55.850 DEBUG:teuthology.orchestra.run.smithi186:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-09-30T00:57:55.857 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:57:55.864 INFO:tasks.workunit.client.0.smithi186.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-09-30T00:58:39.686 INFO:tasks.workunit.client.0.smithi186.stderr:Note: switching to '93dce5e17918f8aca65993a7a2dc40233a47b481'. 2023-09-30T00:58:39.686 INFO:tasks.workunit.client.0.smithi186.stderr: 2023-09-30T00:58:39.687 INFO:tasks.workunit.client.0.smithi186.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-09-30T00:58:39.687 INFO:tasks.workunit.client.0.smithi186.stderr:changes and commit them, and you can discard any commits you make in this 2023-09-30T00:58:39.687 INFO:tasks.workunit.client.0.smithi186.stderr:state without impacting any branches by switching back to a branch. 2023-09-30T00:58:39.687 INFO:tasks.workunit.client.0.smithi186.stderr: 2023-09-30T00:58:39.687 INFO:tasks.workunit.client.0.smithi186.stderr:If you want to create a new branch to retain commits you create, you may 2023-09-30T00:58:39.687 INFO:tasks.workunit.client.0.smithi186.stderr:do so (now or later) by using -c with the switch command. Example: 2023-09-30T00:58:39.687 INFO:tasks.workunit.client.0.smithi186.stderr: 2023-09-30T00:58:39.687 INFO:tasks.workunit.client.0.smithi186.stderr: git switch -c 2023-09-30T00:58:39.687 INFO:tasks.workunit.client.0.smithi186.stderr: 2023-09-30T00:58:39.687 INFO:tasks.workunit.client.0.smithi186.stderr:Or undo this operation with: 2023-09-30T00:58:39.688 INFO:tasks.workunit.client.0.smithi186.stderr: 2023-09-30T00:58:39.688 INFO:tasks.workunit.client.0.smithi186.stderr: git switch - 2023-09-30T00:58:39.688 INFO:tasks.workunit.client.0.smithi186.stderr: 2023-09-30T00:58:39.688 INFO:tasks.workunit.client.0.smithi186.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-09-30T00:58:39.688 INFO:tasks.workunit.client.0.smithi186.stderr: 2023-09-30T00:58:39.688 INFO:tasks.workunit.client.0.smithi186.stderr:HEAD is now at 93dce5e1791 Merge branch 'wip-ksirivad-quincy-backport-52442' of https://github.com/kamoltat/ceph into wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:58:39.691 DEBUG:teuthology.orchestra.run.smithi186:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-09-30T00:58:39.740 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-09-30T00:58:39.740 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-09-30T00:58:39.745 INFO:tasks.workunit:Running workunits matching misc on client.0... 2023-09-30T00:58:39.746 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2023-09-30T00:58:39.746 DEBUG:teuthology.orchestra.run.smithi186:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=93dce5e17918f8aca65993a7a2dc40233a47b481 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2023-09-30T00:58:39.773 INFO:tasks.workunit.client.0.smithi186.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-09-30T00:58:39.789 INFO:tasks.workunit.client.0.smithi186.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-09-30T00:58:39.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-09-30T00:58:39.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-09-30T00:58:39.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: export PYTHONWARNINGS=ignore 2023-09-30T00:58:39.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: PYTHONWARNINGS=ignore 2023-09-30T00:58:39.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: export CEPH_CONF=/dev/null 2023-09-30T00:58:39.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: CEPH_CONF=/dev/null 2023-09-30T00:58:39.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: unset CEPH_ARGS 2023-09-30T00:58:39.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: main: local code 2023-09-30T00:58:39.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: main: run td/mclock-config 2023-09-30T00:58:39.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2023-09-30T00:58:39.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2023-09-30T00:58:39.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2023-09-30T00:58:39.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2023-09-30T00:58:39.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2023-09-30T00:58:39.791 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2023-09-30T00:58:39.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none ' 2023-09-30T00:58:39.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2023-09-30T00:58:39.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2023-09-30T00:58:39.792 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2023-09-30T00:58:39.792 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-09-30T00:58:39.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2023-09-30T00:58:39.795 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_profile_builtin_to_custom 2023-09-30T00:58:39.795 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_profile_custom_to_builtin 2023-09-30T00:58:39.795 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_profile_disallow_builtin_params_modify 2023-09-30T00:58:39.795 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_profile_disallow_builtin_params_override 2023-09-30T00:58:39.795 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_recovery_limit_adjustment_mclock' 2023-09-30T00:58:39.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2023-09-30T00:58:39.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2023-09-30T00:58:39.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2023-09-30T00:58:39.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2023-09-30T00:58:39.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T00:58:39.796 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T00:58:39.796 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T00:58:39.796 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T00:58:39.796 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T00:58:39.796 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T00:58:39.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T00:58:39.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T00:58:39.799 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T00:58:39.799 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T00:58:39.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T00:58:39.800 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T00:58:39.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T00:58:39.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T00:58:39.803 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T00:58:39.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:58:39.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T00:58:39.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T00:58:39.805 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:58:39.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:58:39.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T00:58:39.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T00:58:39.810 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T00:58:39.810 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:58:39.810 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:58:39.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T00:58:39.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T00:58:39.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T00:58:39.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2023-09-30T00:58:39.814 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T00:58:39.814 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:58:39.814 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:58:39.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26283 2023-09-30T00:58:39.816 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T00:58:39.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-09-30T00:58:39.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-09-30T00:58:39.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T00:58:39.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2023-09-30T00:58:39.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2023-09-30T00:58:39.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2023-09-30T00:58:39.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2023-09-30T00:58:39.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2023-09-30T00:58:39.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2023-09-30T00:58:39.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T00:58:39.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T00:58:39.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T00:58:39.819 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T00:58:39.819 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T00:58:39.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T00:58:39.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T00:58:39.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T00:58:39.821 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T00:58:39.822 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T00:58:39.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T00:58:39.824 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T00:58:39.826 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T00:58:39.826 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T00:58:39.826 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T00:58:39.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:58:39.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T00:58:39.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T00:58:39.828 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:58:39.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:58:39.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T00:58:39.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T00:58:39.834 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T00:58:39.834 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:58:39.834 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:58:39.834 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T00:58:39.836 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T00:58:39.836 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T00:58:39.836 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2023-09-30T00:58:39.838 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T00:58:39.838 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:58:39.838 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:58:39.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26283 2023-09-30T00:58:39.841 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T00:58:39.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T00:58:39.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T00:58:39.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2023-09-30T00:58:39.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2023-09-30T00:58:39.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2023-09-30T00:58:39.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T00:58:39.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T00:58:39.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T00:58:39.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2023-09-30T00:58:39.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2023-09-30T00:58:40.310 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T00:58:40.310 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T00:58:40.310 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T00:58:40.311 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T00:58:40.311 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:58:40.311 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:58:40.312 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T00:58:40.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T00:58:40.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T00:58:40.610 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T00:58:40.610 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T00:58:40.610 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T00:58:40.610 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T00:58:40.610 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T00:58:40.611 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T00:58:40.611 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T00:58:40.611 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T00:58:40.612 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:58:40.612 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:58:40.612 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:58:40.612 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T00:58:40.613 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T00:58:40.613 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get fsid 2023-09-30T00:58:40.697 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T00:58:40.697 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T00:58:40.698 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T00:58:40.698 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T00:58:40.699 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T00:58:40.699 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T00:58:40.699 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T00:58:40.699 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T00:58:40.699 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:58:40.699 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:58:40.700 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:58:40.700 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T00:58:40.700 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T00:58:40.700 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get mon_host 2023-09-30T00:58:40.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2023-09-30T00:58:40.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2023-09-30T00:58:40.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T00:58:40.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T00:58:40.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T00:58:40.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2023-09-30T00:58:40.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T00:58:41.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T00:58:41.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T00:58:41.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T00:58:41.684 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T00:58:41.684 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:58:41.684 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:58:41.684 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T00:58:41.685 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T00:58:41.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T00:58:41.848 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2023-09-30T00:58:41.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2023-09-30T00:58:41.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T00:58:41.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T00:58:41.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T00:58:41.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2023-09-30T00:58:41.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2023-09-30T00:58:41.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T00:58:41.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T00:58:41.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T00:58:41.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2023-09-30T00:58:41.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2023-09-30T00:58:41.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T00:58:41.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T00:58:41.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2023-09-30T00:58:41.850 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T00:58:41.850 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T00:58:41.850 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T00:58:41.851 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T00:58:41.851 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:58:41.852 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:58:41.852 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T00:58:41.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T00:58:41.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T00:58:41.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T00:58:41.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T00:58:41.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2023-09-30T00:58:41.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2023-09-30T00:58:41.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T00:58:41.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T00:58:41.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T00:58:41.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T00:58:41.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T00:58:41.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2023-09-30T00:58:41.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2023-09-30T00:58:41.855 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T00:58:41.856 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fb103fee-7546-4240-a322-fd978f83f7e6 2023-09-30T00:58:41.856 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 fb103fee-7546-4240-a322-fd978f83f7e6 2023-09-30T00:58:41.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fb103fee-7546-4240-a322-fd978f83f7e6' 2023-09-30T00:58:41.857 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T00:58:41.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBchdloK0lNBAA+UmLb0UKL5jOQiXtW/edWg== 2023-09-30T00:58:41.876 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBchdloK0lNBAA+UmLb0UKL5jOQiXtW/edWg=="}' 2023-09-30T00:58:41.876 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fb103fee-7546-4240-a322-fd978f83f7e6 -i td/mclock-config/0/new.json 2023-09-30T00:58:42.261 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T00:58:42.279 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2023-09-30T00:58:42.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDBchdloK0lNBAA+UmLb0UKL5jOQiXtW/edWg== --osd-uuid fb103fee-7546-4240-a322-fd978f83f7e6 2023-09-30T00:58:42.346 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:42.342+0000 7f96f5d343c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:58:42.414 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:42.410+0000 7f96f5d343c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:58:42.416 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:42.414+0000 7f96f5d343c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:58:42.417 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:42.414+0000 7f96f5d343c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T00:58:42.418 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:42.414+0000 7f96f5d343c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T00:58:42.418 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:42.414+0000 7f96f5d343c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T00:58:42.453 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:42.450+0000 7f96f5d343c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2023-09-30T00:58:48.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2023-09-30T00:58:48.398 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T00:58:48.399 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T00:58:48.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T00:58:48.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T00:58:48.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T00:58:48.625 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T00:58:48.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2023-09-30T00:58:48.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T00:58:48.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T00:58:48.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T00:58:48.649 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:48.646+0000 7fe554cde3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:58:48.657 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:48.654+0000 7fe554cde3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:58:48.659 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:48.654+0000 7fe554cde3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:58:48.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T00:58:48.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T00:58:48.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T00:58:48.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T00:58:48.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T00:58:48.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:58:48.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T00:58:48.861 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T00:58:48.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:58:48.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:58:49.075 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:58:49.226 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:49.222+0000 7fe554cde3c0 -1 Falling back to public interface 2023-09-30T00:58:50.076 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:58:50.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:58:50.077 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T00:58:50.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T00:58:50.078 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:58:50.078 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:58:50.505 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:58:51.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:58:51.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:58:51.508 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T00:58:51.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T00:58:51.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:58:51.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:58:51.777 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:58:51.774+0000 7fe554cde3c0 -1 osd.0 0 log_to_monitors true 2023-09-30T00:58:52.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:58:53.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:58:53.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:58:53.047 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T00:58:53.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T00:58:53.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:58:53.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:58:53.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:58:54.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:58:54.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:58:54.509 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T00:58:54.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T00:58:54.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:58:54.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:58:54.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:58:55.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:58:55.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:58:55.902 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T00:58:55.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T00:58:55.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:58:55.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:58:56.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:58:57.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:58:57.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:58:57.306 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T00:58:57.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T00:58:57.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:58:57.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:58:57.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:58:58.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:58:58.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:58:58.833 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T00:58:58.833 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T00:58:58.834 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:58:58.834 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:58:59.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:00.220 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:00.220 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:00.221 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T00:59:00.221 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T00:59:00.221 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:00.221 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:00.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:01.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:01.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:01.624 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T00:59:01.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T00:59:01.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:01.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:02.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:03.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:03.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:03.032 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T00:59:03.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T00:59:03.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:03.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:03.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:04.882 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T00:59:04.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T00:59:04.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:04.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:05.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:06.320 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:06.321 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:06.321 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T00:59:06.321 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T00:59:06.321 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:06.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:07.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:08.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:08.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:08.306 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T00:59:08.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T00:59:08.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:08.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:09.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:10.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:10.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:10.163 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T00:59:10.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T00:59:10.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:10.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:10.835 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:11.837 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:11.837 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:11.837 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T00:59:11.837 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T00:59:11.838 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:11.838 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:12.255 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:13.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:13.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:13.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T00:59:13.258 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T00:59:13.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:13.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:14.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:15.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:15.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:15.448 INFO:tasks.workunit.client.0.smithi186.stdout:17 2023-09-30T00:59:15.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-09-30T00:59:15.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:15.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:15.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:16.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:16.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:16.859 INFO:tasks.workunit.client.0.smithi186.stdout:18 2023-09-30T00:59:16.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-09-30T00:59:16.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:16.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:17.344 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/27341,v1:127.0.0.1:6801/27341] [v2:127.0.0.1:6802/27341,v1:127.0.0.1:6803/27341] exists,up fb103fee-7546-4240-a322-fd978f83f7e6 2023-09-30T00:59:17.344 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T00:59:17.344 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T00:59:17.344 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T00:59:17.345 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2023-09-30T00:59:17.345 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2023-09-30T00:59:17.346 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T00:59:17.346 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T00:59:17.346 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T00:59:17.347 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:17.347 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:17.347 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:17.347 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T00:59:17.347 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T00:59:17.348 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_max_backfills 2023-09-30T00:59:17.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2023-09-30T00:59:17.439 INFO:tasks.workunit.client.0.smithi186.stdout:osd_max_backfills: 1 2023-09-30T00:59:17.439 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2023-09-30T00:59:17.439 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2023-09-30T00:59:18.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2023-09-30T00:59:20.019 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2023-09-30T00:59:20.019 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2023-09-30T00:59:20.020 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T00:59:20.020 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T00:59:20.020 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T00:59:20.021 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:20.021 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:20.021 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:20.021 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T00:59:20.022 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T00:59:20.022 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_max_backfills 2023-09-30T00:59:20.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2023-09-30T00:59:20.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2023-09-30T00:59:20.119 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2023-09-30T00:59:20.119 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2023-09-30T00:59:20.120 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T00:59:20.120 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T00:59:20.120 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T00:59:20.120 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:20.121 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:20.121 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:20.121 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T00:59:20.121 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T00:59:20.121 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok dump_recovery_reservations 2023-09-30T00:59:20.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2023-09-30T00:59:20.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2023-09-30T00:59:20.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2023-09-30T00:59:20.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2023-09-30T00:59:20.215 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T00:59:20.215 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T00:59:20.216 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T00:59:20.216 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:20.216 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:20.216 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:20.217 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T00:59:20.217 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T00:59:20.217 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok dump_recovery_reservations 2023-09-30T00:59:20.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2023-09-30T00:59:20.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2023-09-30T00:59:20.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2023-09-30T00:59:20.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2023-09-30T00:59:21.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2023-09-30T00:59:23.333 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2023-09-30T00:59:23.333 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2023-09-30T00:59:23.334 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T00:59:23.334 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T00:59:23.334 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T00:59:23.335 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:23.335 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:23.335 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:23.336 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T00:59:23.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T00:59:23.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_max_backfills 2023-09-30T00:59:23.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2023-09-30T00:59:23.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2023-09-30T00:59:23.436 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2023-09-30T00:59:23.436 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2023-09-30T00:59:23.437 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T00:59:23.437 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T00:59:23.437 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T00:59:23.438 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:23.438 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:23.438 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:23.438 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T00:59:23.438 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T00:59:23.439 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok dump_recovery_reservations 2023-09-30T00:59:23.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2023-09-30T00:59:23.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2023-09-30T00:59:23.527 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2023-09-30T00:59:23.527 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2023-09-30T00:59:23.528 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T00:59:23.528 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T00:59:23.528 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T00:59:23.529 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:23.529 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:23.529 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:23.529 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T00:59:23.529 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T00:59:23.529 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok dump_recovery_reservations 2023-09-30T00:59:23.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2023-09-30T00:59:23.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2023-09-30T00:59:23.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2023-09-30T00:59:23.621 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T00:59:23.621 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T00:59:23.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T00:59:23.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T00:59:23.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T00:59:23.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T00:59:23.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2023-09-30T00:59:24.375 INFO:tasks.workunit.client.0.smithi186.stderr:osd.0 is already down. 2023-09-30T00:59:24.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2023-09-30T00:59:24.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2023-09-30T00:59:24.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T00:59:24.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T00:59:24.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T00:59:24.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:24.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T00:59:24.387 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T00:59:24.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:24.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2023-09-30T00:59:24.813 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6800/27341,v1:127.0.0.1:6801/27341] [v2:127.0.0.1:6802/27341,v1:127.0.0.1:6803/27341] exists fb103fee-7546-4240-a322-fd978f83f7e6 2023-09-30T00:59:24.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T00:59:24.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T00:59:24.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T00:59:24.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2023-09-30T00:59:24.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2023-09-30T00:59:24.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-09-30T00:59:24.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-09-30T00:59:24.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-09-30T00:59:24.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2023-09-30T00:59:24.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2023-09-30T00:59:24.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T00:59:24.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T00:59:24.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T00:59:24.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2023-09-30T00:59:24.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2023-09-30T00:59:24.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-09-30T00:59:24.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-09-30T00:59:24.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2023-09-30T00:59:24.816 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-09-30T00:59:24.819 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T00:59:24.819 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T00:59:24.819 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T00:59:24.819 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:24.819 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:24.820 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T00:59:24.821 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T00:59:24.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-09-30T00:59:24.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2023-09-30T00:59:24.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2023-09-30T00:59:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T00:59:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T00:59:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T00:59:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T00:59:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-09-30T00:59:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2023-09-30T00:59:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2023-09-30T00:59:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-09-30T00:59:24.824 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T00:59:24.824 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2023-09-30T00:59:24.824 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2023-09-30T00:59:24.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-09-30T00:59:24.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-09-30T00:59:24.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-09-30T00:59:24.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-09-30T00:59:24.843 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:24.837+0000 7f615c30e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:59:24.846 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:24.841+0000 7f615c30e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:59:24.848 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:24.845+0000 7f615c30e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:59:25.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-09-30T00:59:25.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T00:59:25.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T00:59:25.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T00:59:25.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T00:59:25.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:25.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T00:59:25.188 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T00:59:25.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:25.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:25.394 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:25.389+0000 7f615c30e3c0 -1 Falling back to public interface 2023-09-30T00:59:25.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:26.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:26.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:26.555 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T00:59:26.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T00:59:26.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:26.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:26.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:27.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:27.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:27.963 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T00:59:27.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T00:59:27.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:27.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:28.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:28.521 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:28.517+0000 7f615c30e3c0 -1 osd.0 6 log_to_monitors true 2023-09-30T00:59:29.440 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:29.441 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:29.441 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T00:59:29.442 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T00:59:29.442 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:29.442 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:29.903 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6800/28640,v1:127.0.0.1:6801/28640] [v2:127.0.0.1:6802/28640,v1:127.0.0.1:6803/28640] exists,up fb103fee-7546-4240-a322-fd978f83f7e6 2023-09-30T00:59:29.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T00:59:29.904 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T00:59:29.904 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T00:59:29.904 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2023-09-30T00:59:29.905 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T00:59:29.905 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T00:59:29.905 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T00:59:29.905 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2023-09-30T00:59:29.906 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:29.906 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:29.906 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:29.907 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T00:59:29.907 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T00:59:29.907 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_max_backfills 2023-09-30T00:59:30.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2023-09-30T00:59:30.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2023-09-30T00:59:30.002 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2023-09-30T00:59:30.002 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2023-09-30T00:59:30.003 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T00:59:30.003 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T00:59:30.003 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T00:59:30.004 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:30.004 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:30.004 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:30.005 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T00:59:30.005 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T00:59:30.005 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok dump_recovery_reservations 2023-09-30T00:59:30.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2023-09-30T00:59:30.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2023-09-30T00:59:30.096 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T00:59:30.097 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2023-09-30T00:59:30.097 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T00:59:30.097 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T00:59:30.097 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2023-09-30T00:59:30.097 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:30.098 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:30.098 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:30.098 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T00:59:30.098 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T00:59:30.098 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok dump_recovery_reservations 2023-09-30T00:59:30.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2023-09-30T00:59:30.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2023-09-30T00:59:30.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2023-09-30T00:59:30.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T00:59:30.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T00:59:30.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T00:59:30.187 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T00:59:30.187 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T00:59:30.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T00:59:30.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T00:59:30.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T00:59:30.298 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T00:59:30.299 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T00:59:30.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T00:59:30.301 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T00:59:30.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T00:59:30.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T00:59:30.304 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T00:59:30.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:59:30.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T00:59:30.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T00:59:30.307 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:59:30.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:59:30.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T00:59:30.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T00:59:30.321 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T00:59:30.322 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:30.322 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:30.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T00:59:30.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T00:59:30.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T00:59:30.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2023-09-30T00:59:30.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T00:59:30.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T00:59:30.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T00:59:30.326 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T00:59:30.327 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T00:59:30.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T00:59:30.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T00:59:30.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T00:59:30.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T00:59:30.330 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T00:59:30.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T00:59:30.332 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T00:59:30.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T00:59:30.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T00:59:30.334 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T00:59:30.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:59:30.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T00:59:30.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T00:59:30.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:59:30.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:59:30.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T00:59:30.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T00:59:30.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T00:59:30.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:30.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:30.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T00:59:30.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T00:59:30.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T00:59:30.348 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2023-09-30T00:59:30.348 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2023-09-30T00:59:30.348 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2023-09-30T00:59:30.348 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2023-09-30T00:59:30.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T00:59:30.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T00:59:30.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T00:59:30.349 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T00:59:30.349 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T00:59:30.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T00:59:30.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T00:59:30.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T00:59:30.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T00:59:30.350 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T00:59:30.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T00:59:30.353 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T00:59:30.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T00:59:30.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T00:59:30.355 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T00:59:30.357 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:59:30.357 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T00:59:30.357 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T00:59:30.358 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:59:30.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:59:30.361 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T00:59:30.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T00:59:30.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T00:59:30.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:30.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:30.364 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T00:59:30.365 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T00:59:30.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T00:59:30.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2023-09-30T00:59:30.368 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T00:59:30.368 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:30.368 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:30.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26283 2023-09-30T00:59:30.371 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T00:59:30.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T00:59:30.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T00:59:30.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2023-09-30T00:59:30.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2023-09-30T00:59:30.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2023-09-30T00:59:30.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2023-09-30T00:59:30.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2023-09-30T00:59:30.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T00:59:30.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T00:59:30.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T00:59:30.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2023-09-30T00:59:30.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2023-09-30T00:59:30.744 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T00:59:30.745 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T00:59:30.745 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T00:59:30.746 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T00:59:30.746 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:30.746 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:30.746 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T00:59:30.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T00:59:31.043 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T00:59:31.045 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T00:59:31.045 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T00:59:31.045 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T00:59:31.045 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T00:59:31.045 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T00:59:31.046 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T00:59:31.046 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T00:59:31.046 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T00:59:31.047 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:31.047 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:31.047 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:31.048 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T00:59:31.049 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T00:59:31.049 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get fsid 2023-09-30T00:59:31.141 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T00:59:31.142 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T00:59:31.142 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T00:59:31.142 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T00:59:31.142 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T00:59:31.143 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T00:59:31.143 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T00:59:31.143 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T00:59:31.143 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T00:59:31.143 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:31.144 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:31.144 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T00:59:31.144 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T00:59:31.145 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get mon_host 2023-09-30T00:59:31.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2023-09-30T00:59:31.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2023-09-30T00:59:31.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T00:59:31.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T00:59:31.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T00:59:31.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2023-09-30T00:59:31.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T00:59:32.119 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T00:59:32.119 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T00:59:32.119 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T00:59:32.120 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T00:59:32.120 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:32.120 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:32.120 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T00:59:32.121 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T00:59:32.122 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T00:59:32.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2023-09-30T00:59:32.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2023-09-30T00:59:32.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T00:59:32.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T00:59:32.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T00:59:32.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2023-09-30T00:59:32.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2023-09-30T00:59:32.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T00:59:32.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T00:59:32.159 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T00:59:32.159 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2023-09-30T00:59:32.159 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2023-09-30T00:59:32.159 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T00:59:32.159 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T00:59:32.159 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2023-09-30T00:59:32.159 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T00:59:32.159 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T00:59:32.159 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T00:59:32.161 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T00:59:32.161 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T00:59:32.161 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T00:59:32.161 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T00:59:32.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T00:59:32.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T00:59:32.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T00:59:32.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T00:59:32.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2023-09-30T00:59:32.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2023-09-30T00:59:32.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T00:59:32.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T00:59:32.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T00:59:32.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T00:59:32.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T00:59:32.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2023-09-30T00:59:32.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2023-09-30T00:59:32.164 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T00:59:32.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6fe2faa3-6b8b-4665-9061-1b5527164df2 2023-09-30T00:59:32.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6fe2faa3-6b8b-4665-9061-1b5527164df2' 2023-09-30T00:59:32.164 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 6fe2faa3-6b8b-4665-9061-1b5527164df2 2023-09-30T00:59:32.164 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T00:59:32.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0chdlTMPrChAAPUm4nVqE0Rcr5S6n4Ywt3Q== 2023-09-30T00:59:32.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0chdlTMPrChAAPUm4nVqE0Rcr5S6n4Ywt3Q=="}' 2023-09-30T00:59:32.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6fe2faa3-6b8b-4665-9061-1b5527164df2 -i td/mclock-config/0/new.json 2023-09-30T00:59:32.536 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T00:59:32.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2023-09-30T00:59:32.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQD0chdlTMPrChAAPUm4nVqE0Rcr5S6n4Ywt3Q== --osd-uuid 6fe2faa3-6b8b-4665-9061-1b5527164df2 2023-09-30T00:59:32.581 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:32.577+0000 7f8f624343c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:59:32.584 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:32.581+0000 7f8f624343c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:59:32.585 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:32.581+0000 7f8f624343c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:59:32.586 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:32.581+0000 7f8f624343c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T00:59:32.586 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:32.581+0000 7f8f624343c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T00:59:32.586 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:32.581+0000 7f8f624343c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T00:59:32.651 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:32.645+0000 7f8f624343c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2023-09-30T00:59:38.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2023-09-30T00:59:38.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T00:59:38.899 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T00:59:38.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T00:59:38.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T00:59:39.461 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T00:59:39.462 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T00:59:39.462 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2023-09-30T00:59:39.462 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T00:59:39.462 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T00:59:39.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T00:59:39.483 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:39.481+0000 7f25750483c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:59:39.485 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:39.481+0000 7f25750483c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:59:39.487 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:39.481+0000 7f25750483c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T00:59:39.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T00:59:39.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T00:59:39.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T00:59:39.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T00:59:39.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T00:59:39.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:39.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T00:59:39.900 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T00:59:39.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:39.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:40.022 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:40.017+0000 7f25750483c0 -1 Falling back to public interface 2023-09-30T00:59:40.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:41.271 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:41.271 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:41.271 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T00:59:41.271 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T00:59:41.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:41.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:41.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:42.552 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T00:59:42.549+0000 7f25750483c0 -1 osd.0 0 log_to_monitors true 2023-09-30T00:59:42.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:42.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:42.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T00:59:42.661 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T00:59:42.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:42.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:43.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:44.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:44.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:44.282 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T00:59:44.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T00:59:44.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:44.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:44.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:45.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:45.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:45.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T00:59:45.749 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T00:59:45.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:45.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:46.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:47.233 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:47.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:47.234 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T00:59:47.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T00:59:47.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:47.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:47.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:48.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:48.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:48.659 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T00:59:48.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T00:59:48.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:48.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:49.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:50.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:50.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:50.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T00:59:50.063 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T00:59:50.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:50.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:50.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:51.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:51.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:51.564 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T00:59:51.564 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T00:59:51.564 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:51.564 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:52.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:53.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:53.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:53.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T00:59:53.016 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T00:59:53.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:53.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:53.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:54.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:54.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:54.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T00:59:54.804 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T00:59:54.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:54.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:55.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:56.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:56.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:56.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T00:59:56.488 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T00:59:56.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:56.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:57.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T00:59:58.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T00:59:58.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T00:59:58.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T00:59:58.203 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T00:59:58.204 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T00:59:58.204 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T00:59:59.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:00.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:00.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:00.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:00:00.245 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:00:00.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:00.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:01.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:02.067 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:02.067 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:02.067 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:00:02.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:00:02.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:02.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:02.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:03.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:03.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:03.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:00:03.845 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:00:03.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:03.846 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:04.369 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/30380,v1:127.0.0.1:6803/30380] [v2:127.0.0.1:6804/30380,v1:127.0.0.1:6805/30380] exists,up 6fe2faa3-6b8b-4665-9061-1b5527164df2 2023-09-30T01:00:04.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:00:04.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:00:04.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:00:04.370 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2023-09-30T01:00:04.794 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2023-09-30T01:00:04.794 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2023-09-30T01:00:04.795 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2023-09-30T01:00:04.796 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2023-09-30T01:00:04.796 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:00:04.796 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:00:04.797 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:04.797 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:04.797 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:04.797 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:00:04.798 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2023-09-30T01:00:04.798 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_profile 2023-09-30T01:00:04.888 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2023-09-30T01:00:04.888 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2023-09-30T01:00:04.889 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2023-09-30T01:00:04.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2023-09-30T01:00:04.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2023-09-30T01:00:04.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2023-09-30T01:00:05.020 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:00:05.020 INFO:tasks.workunit.client.0.smithi186.stdout: "success": "" 2023-09-30T01:00:05.020 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:00:05.032 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2023-09-30T01:00:05.032 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2023-09-30T01:00:05.032 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:00:05.032 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:00:05.033 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:05.033 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:05.033 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:05.033 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:00:05.034 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2023-09-30T01:00:05.034 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_profile 2023-09-30T01:00:05.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2023-09-30T01:00:05.148 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2023-09-30T01:00:05.148 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2023-09-30T01:00:05.149 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2023-09-30T01:00:05.149 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2023-09-30T01:00:05.150 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2023-09-30T01:00:05.150 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2023-09-30T01:00:05.150 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2023-09-30T01:00:05.150 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:00:05.150 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:00:05.150 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:05.151 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:05.151 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:05.151 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:00:05.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2023-09-30T01:00:05.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2023-09-30T01:00:05.219 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2023-09-30T01:00:05.220 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2023-09-30T01:00:05.220 INFO:tasks.workunit.client.0.smithi186.stdout:client_res = 0.300000 2023-09-30T01:00:05.220 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2023-09-30T01:00:05.220 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2023-09-30T01:00:05.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2023-09-30T01:00:05.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2023-09-30T01:00:05.223 INFO:tasks.workunit.client.0.smithi186.stdout:client_res_new = .400000 2023-09-30T01:00:05.223 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2023-09-30T01:00:05.696 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2023-09-30T01:00:06.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2023-09-30T01:00:06.091 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2023-09-30T01:00:06.091 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2023-09-30T01:00:06.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2023-09-30T01:00:06.094 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2023-09-30T01:00:06.094 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2023-09-30T01:00:06.095 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2023-09-30T01:00:06.095 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:00:06.095 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:00:06.095 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:06.095 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:06.096 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:06.096 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:00:06.097 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2023-09-30T01:00:06.097 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2023-09-30T01:00:06.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2023-09-30T01:00:06.188 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2023-09-30T01:00:06.188 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2023-09-30T01:00:06.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2023-09-30T01:00:06.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2023-09-30T01:00:06.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:00:06.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:00:06.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:00:06.191 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:00:06.191 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:00:06.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:00:06.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:00:06.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:00:06.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:00:06.302 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:00:06.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:00:06.305 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:00:06.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:00:06.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:00:06.307 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:00:06.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:06.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:00:06.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:00:06.310 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:06.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:00:06.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:00:06.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:00:06.322 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:00:06.322 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:06.322 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:06.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:00:06.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:00:06.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:00:06.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2023-09-30T01:00:06.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:00:06.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:00:06.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:00:06.325 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:00:06.325 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:00:06.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:00:06.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:00:06.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:00:06.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:00:06.329 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:00:06.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:00:06.331 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:00:06.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:00:06.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:00:06.334 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:00:06.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:06.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:00:06.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:00:06.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:06.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:00:06.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:00:06.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:00:06.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:00:06.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:06.343 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:06.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:00:06.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:00:06.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:00:06.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2023-09-30T01:00:06.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2023-09-30T01:00:06.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2023-09-30T01:00:06.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2023-09-30T01:00:06.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:00:06.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:00:06.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:00:06.346 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:00:06.346 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:00:06.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:00:06.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:00:06.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:00:06.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:00:06.350 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:00:06.351 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:00:06.352 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:00:06.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:00:06.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:00:06.355 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:00:06.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:06.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:00:06.357 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:00:06.357 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:06.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:00:06.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:00:06.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:00:06.363 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:00:06.363 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:06.363 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:06.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:00:06.365 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:00:06.365 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:00:06.365 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2023-09-30T01:00:06.368 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:00:06.368 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:06.368 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:06.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26283 2023-09-30T01:00:06.370 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:00:06.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:00:06.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:00:06.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2023-09-30T01:00:06.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2023-09-30T01:00:06.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2023-09-30T01:00:06.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2023-09-30T01:00:06.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2023-09-30T01:00:06.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2023-09-30T01:00:06.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:00:06.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:00:06.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:00:06.373 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:00:06.373 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:00:06.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:00:06.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:00:06.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:00:06.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:00:06.378 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:00:06.379 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:00:06.380 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:00:06.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:00:06.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:00:06.382 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:00:06.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:06.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:00:06.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:00:06.385 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:06.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:00:06.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:00:06.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:00:06.391 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:00:06.391 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:06.392 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:06.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:00:06.393 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:00:06.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:00:06.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2023-09-30T01:00:06.396 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:00:06.396 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:06.397 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:06.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26283 2023-09-30T01:00:06.399 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:00:06.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:00:06.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:00:06.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2023-09-30T01:00:06.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2023-09-30T01:00:06.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2023-09-30T01:00:06.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:00:06.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:00:06.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:00:06.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2023-09-30T01:00:06.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2023-09-30T01:00:06.760 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:00:06.760 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:00:06.760 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:00:06.761 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:00:06.761 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:06.761 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:06.761 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:00:06.761 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:00:07.067 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:00:07.068 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:00:07.068 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:00:07.068 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:00:07.068 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:00:07.069 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:00:07.069 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:00:07.069 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:00:07.069 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:00:07.070 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:07.070 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:07.071 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:07.073 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T01:00:07.073 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:00:07.073 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get fsid 2023-09-30T01:00:07.156 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:00:07.156 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:00:07.156 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:00:07.156 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:00:07.156 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:00:07.157 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:00:07.157 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:00:07.157 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:00:07.158 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:07.158 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:07.158 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:07.158 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T01:00:07.159 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:00:07.159 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get mon_host 2023-09-30T01:00:07.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2023-09-30T01:00:07.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2023-09-30T01:00:07.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:00:07.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:00:07.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:00:07.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2023-09-30T01:00:07.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:00:08.123 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:00:08.123 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:00:08.123 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:00:08.123 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:00:08.124 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:08.124 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:08.124 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:00:08.125 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:00:08.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:00:08.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2023-09-30T01:00:08.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2023-09-30T01:00:08.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:00:08.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:00:08.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:00:08.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2023-09-30T01:00:08.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2023-09-30T01:00:08.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:00:08.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:00:08.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:00:08.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2023-09-30T01:00:08.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2023-09-30T01:00:08.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:00:08.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:00:08.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2023-09-30T01:00:08.164 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:00:08.164 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:00:08.164 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:00:08.164 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:00:08.165 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:08.165 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:08.165 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:00:08.166 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:00:08.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:00:08.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:00:08.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:00:08.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2023-09-30T01:00:08.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2023-09-30T01:00:08.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:00:08.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:00:08.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:00:08.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:00:08.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:00:08.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2023-09-30T01:00:08.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2023-09-30T01:00:08.168 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:00:08.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9ac15462-af19-49a5-a296-e51012e63465 2023-09-30T01:00:08.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9ac15462-af19-49a5-a296-e51012e63465' 2023-09-30T01:00:08.169 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 9ac15462-af19-49a5-a296-e51012e63465 2023-09-30T01:00:08.169 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:00:08.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAYcxdlEKcTCxAAENcJ0umcGT1nkXNG9dx8Ng== 2023-09-30T01:00:08.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAYcxdlEKcTCxAAENcJ0umcGT1nkXNG9dx8Ng=="}' 2023-09-30T01:00:08.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9ac15462-af19-49a5-a296-e51012e63465 -i td/mclock-config/0/new.json 2023-09-30T01:00:08.542 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:00:08.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2023-09-30T01:00:08.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAYcxdlEKcTCxAAENcJ0umcGT1nkXNG9dx8Ng== --osd-uuid 9ac15462-af19-49a5-a296-e51012e63465 2023-09-30T01:00:08.583 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:08.577+0000 7f9e41e543c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:08.586 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:08.581+0000 7f9e41e543c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:08.587 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:08.585+0000 7f9e41e543c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:08.588 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:08.585+0000 7f9e41e543c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:00:08.588 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:08.585+0000 7f9e41e543c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:00:08.589 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:08.585+0000 7f9e41e543c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:00:08.665 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:08.661+0000 7f9e41e543c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2023-09-30T01:00:14.878 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2023-09-30T01:00:14.878 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:00:14.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:00:14.880 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:00:14.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:00:15.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:00:15.444 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:00:15.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2023-09-30T01:00:15.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:00:15.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:00:15.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:00:15.472 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:15.469+0000 7fb8f496a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:15.475 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:15.469+0000 7fb8f496a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:15.476 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:15.473+0000 7fb8f496a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:15.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:00:15.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:00:15.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:00:15.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:00:15.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:00:15.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:15.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:00:15.807 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:00:15.807 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:15.807 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:16.030 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:16.025+0000 7fb8f496a3c0 -1 Falling back to public interface 2023-09-30T01:00:16.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:17.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:17.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:17.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:00:17.228 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:00:17.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:17.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:17.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:18.565 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:18.561+0000 7fb8f496a3c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:00:18.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:18.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:18.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:00:18.624 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:00:18.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:18.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:19.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:20.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:20.120 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:20.120 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:00:20.120 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:00:20.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:20.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:20.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:21.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:21.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:21.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:00:21.560 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:00:21.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:21.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:21.975 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:22.976 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:22.976 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:22.977 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:00:22.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:00:22.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:22.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:23.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:24.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:24.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:24.387 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:00:24.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:00:24.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:24.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:24.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:25.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:25.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:25.792 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:00:25.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:00:25.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:25.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:26.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:27.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:27.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:27.300 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:00:27.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:00:27.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:27.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:27.717 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:28.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:28.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:28.719 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:00:28.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:00:28.720 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:28.720 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:29.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:30.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:30.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:30.155 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:00:30.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:00:30.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:30.156 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:31.149 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:32.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:32.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:32.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:00:32.152 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:00:32.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:32.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:32.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:33.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:33.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:33.694 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:00:33.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:00:33.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:33.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:34.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:35.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:35.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:35.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:00:35.581 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:00:35.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:35.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:36.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:37.348 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:37.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:37.349 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:00:37.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:00:37.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:37.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:38.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:39.132 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:39.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:39.133 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:00:39.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:00:39.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:39.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:39.741 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:00:40.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:00:40.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:40.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:00:40.744 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:00:40.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:00:40.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:41.145 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/32564,v1:127.0.0.1:6803/32564] [v2:127.0.0.1:6804/32564,v1:127.0.0.1:6805/32564] exists,up 9ac15462-af19-49a5-a296-e51012e63465 2023-09-30T01:00:41.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:00:41.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:00:41.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:00:41.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2023-09-30T01:00:41.146 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2023-09-30T01:00:41.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2023-09-30T01:00:41.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2023-09-30T01:00:41.538 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2023-09-30T01:00:41.538 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2023-09-30T01:00:41.538 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:00:41.539 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:00:41.539 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:41.539 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:41.539 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:41.540 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:00:41.540 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2023-09-30T01:00:41.540 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_profile 2023-09-30T01:00:41.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2023-09-30T01:00:41.630 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2023-09-30T01:00:41.630 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2023-09-30T01:00:41.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2023-09-30T01:00:41.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2023-09-30T01:00:41.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2023-09-30T01:00:41.769 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:00:41.769 INFO:tasks.workunit.client.0.smithi186.stdout: "success": "" 2023-09-30T01:00:41.769 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:00:41.781 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2023-09-30T01:00:41.781 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2023-09-30T01:00:41.781 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:00:41.781 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:00:41.782 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:41.782 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:41.782 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:41.783 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:00:41.783 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2023-09-30T01:00:41.783 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_profile 2023-09-30T01:00:41.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2023-09-30T01:00:41.876 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2023-09-30T01:00:41.876 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2023-09-30T01:00:41.876 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2023-09-30T01:00:41.876 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2023-09-30T01:00:41.876 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2023-09-30T01:00:41.880 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2023-09-30T01:00:41.880 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2023-09-30T01:00:41.880 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2023-09-30T01:00:41.880 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:00:41.880 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:00:41.880 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:41.880 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:41.880 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:41.880 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:00:41.881 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2023-09-30T01:00:41.881 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2023-09-30T01:00:41.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2023-09-30T01:00:41.967 INFO:tasks.workunit.client.0.smithi186.stdout:Original client_res for osd.0 = 0.300000 2023-09-30T01:00:41.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2023-09-30T01:00:41.968 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2023-09-30T01:00:41.968 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2023-09-30T01:00:41.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2023-09-30T01:00:41.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2023-09-30T01:00:41.970 INFO:tasks.workunit.client.0.smithi186.stdout:client_res_new = .400000 2023-09-30T01:00:41.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2023-09-30T01:00:42.498 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2023-09-30T01:00:43.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2023-09-30T01:00:43.147 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2023-09-30T01:00:43.147 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2023-09-30T01:00:43.148 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2023-09-30T01:00:43.150 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2023-09-30T01:00:43.150 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2023-09-30T01:00:43.151 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2023-09-30T01:00:43.151 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:00:43.151 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:00:43.151 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:43.151 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:43.152 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:43.152 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:00:43.152 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2023-09-30T01:00:43.152 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2023-09-30T01:00:43.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2023-09-30T01:00:43.244 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2023-09-30T01:00:43.244 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2023-09-30T01:00:43.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2023-09-30T01:00:43.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2023-09-30T01:00:43.376 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:00:43.376 INFO:tasks.workunit.client.0.smithi186.stdout: "success": "" 2023-09-30T01:00:43.376 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:00:43.388 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2023-09-30T01:00:43.388 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2023-09-30T01:00:43.388 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:00:43.388 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:00:43.389 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:43.389 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:43.389 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:43.390 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:00:43.390 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2023-09-30T01:00:43.390 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_profile 2023-09-30T01:00:43.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2023-09-30T01:00:43.483 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2023-09-30T01:00:43.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2023-09-30T01:00:43.484 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2023-09-30T01:00:43.928 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2023-09-30T01:00:43.930 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2023-09-30T01:00:43.930 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2023-09-30T01:00:43.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2023-09-30T01:00:43.934 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2023-09-30T01:00:43.934 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2023-09-30T01:00:43.934 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2023-09-30T01:00:43.934 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:00:43.935 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:00:43.935 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:43.935 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:43.935 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:43.936 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:00:43.936 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2023-09-30T01:00:43.936 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2023-09-30T01:00:44.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2023-09-30T01:00:44.040 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2023-09-30T01:00:44.040 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2023-09-30T01:00:44.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2023-09-30T01:00:44.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2023-09-30T01:00:44.516 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2023-09-30T01:00:49.518 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2023-09-30T01:00:49.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2023-09-30T01:00:49.975 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2023-09-30T01:00:49.975 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2023-09-30T01:00:49.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2023-09-30T01:00:49.979 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2023-09-30T01:00:49.979 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2023-09-30T01:00:49.979 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2023-09-30T01:00:49.979 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:00:49.980 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:00:49.980 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:49.980 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:49.980 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:49.981 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:00:49.981 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2023-09-30T01:00:49.981 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2023-09-30T01:00:50.086 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2023-09-30T01:00:50.087 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2023-09-30T01:00:50.087 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2023-09-30T01:00:50.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2023-09-30T01:00:50.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2023-09-30T01:00:50.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:00:50.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:00:50.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:00:50.090 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:00:50.090 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:00:50.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:00:50.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:00:50.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:00:50.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:00:50.203 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:00:50.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:00:50.205 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:00:50.207 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:00:50.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:00:50.208 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:00:50.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:50.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:00:50.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:00:50.211 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:50.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:00:50.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:00:50.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:00:50.223 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:00:50.223 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:50.224 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:50.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:00:50.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:00:50.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:00:50.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2023-09-30T01:00:50.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:00:50.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:00:50.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:00:50.227 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:00:50.227 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:00:50.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:00:50.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:00:50.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:00:50.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:00:50.231 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:00:50.233 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:00:50.234 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:00:50.236 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:00:50.236 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:00:50.236 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:00:50.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:50.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:00:50.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:00:50.239 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:50.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:00:50.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:00:50.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:00:50.246 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:00:50.246 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:50.246 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:50.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:00:50.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:00:50.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:00:50.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2023-09-30T01:00:50.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2023-09-30T01:00:50.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2023-09-30T01:00:50.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2023-09-30T01:00:50.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:00:50.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:00:50.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:00:50.249 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:00:50.249 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:00:50.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:00:50.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:00:50.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:00:50.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:00:50.254 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:00:50.255 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:00:50.256 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:00:50.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:00:50.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:00:50.259 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:00:50.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:50.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:00:50.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:00:50.262 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:50.263 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:00:50.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:00:50.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:00:50.267 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:00:50.267 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:50.267 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:50.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:00:50.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:00:50.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:00:50.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2023-09-30T01:00:50.272 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:00:50.272 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:50.272 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:50.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26283 2023-09-30T01:00:50.274 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:00:50.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:00:50.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:00:50.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2023-09-30T01:00:50.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2023-09-30T01:00:50.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2023-09-30T01:00:50.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2023-09-30T01:00:50.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2023-09-30T01:00:50.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2023-09-30T01:00:50.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:00:50.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:00:50.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:00:50.277 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:00:50.277 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:00:50.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:00:50.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:00:50.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:00:50.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:00:50.283 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:00:50.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:00:50.285 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:00:50.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:00:50.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:00:50.288 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:00:50.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:50.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:00:50.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:00:50.291 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:00:50.292 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:00:50.294 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:00:50.294 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:00:50.297 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:00:50.297 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:50.297 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:50.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:00:50.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:00:50.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:00:50.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2023-09-30T01:00:50.302 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:00:50.302 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:50.302 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:50.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26283 2023-09-30T01:00:50.305 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:00:50.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:00:50.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:00:50.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2023-09-30T01:00:50.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2023-09-30T01:00:50.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2023-09-30T01:00:50.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:00:50.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:00:50.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:00:50.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2023-09-30T01:00:50.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2023-09-30T01:00:50.686 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:00:50.686 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:00:50.686 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:00:50.687 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:00:50.687 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:50.687 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:50.687 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:00:50.687 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:00:50.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:00:50.985 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:00:50.985 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:00:50.985 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:00:50.985 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:00:50.986 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:00:50.986 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:00:50.986 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:00:50.986 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:00:50.987 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:50.987 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:50.987 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:50.987 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T01:00:50.988 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:00:50.988 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get fsid 2023-09-30T01:00:51.080 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:00:51.080 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:00:51.080 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:00:51.081 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:00:51.081 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:00:51.081 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:00:51.081 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:00:51.081 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:00:51.082 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:00:51.082 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:51.082 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:51.083 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T01:00:51.083 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:00:51.083 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get mon_host 2023-09-30T01:00:51.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2023-09-30T01:00:51.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2023-09-30T01:00:51.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:00:51.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:00:51.169 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:00:51.169 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2023-09-30T01:00:51.169 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:00:52.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:00:52.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:00:52.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:00:52.060 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:00:52.061 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:52.061 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:52.061 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:00:52.062 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:00:52.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:00:52.097 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2023-09-30T01:00:52.097 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2023-09-30T01:00:52.097 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:00:52.097 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:00:52.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:00:52.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2023-09-30T01:00:52.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2023-09-30T01:00:52.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:00:52.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:00:52.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:00:52.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2023-09-30T01:00:52.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2023-09-30T01:00:52.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:00:52.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:00:52.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2023-09-30T01:00:52.099 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:00:52.099 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:00:52.099 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:00:52.099 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:00:52.099 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:00:52.099 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:00:52.100 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:00:52.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:00:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:00:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:00:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:00:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2023-09-30T01:00:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2023-09-30T01:00:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:00:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:00:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:00:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:00:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:00:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2023-09-30T01:00:52.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2023-09-30T01:00:52.102 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:00:52.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bd06baef-4ba4-445d-918f-f168424bbc6a 2023-09-30T01:00:52.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bd06baef-4ba4-445d-918f-f168424bbc6a' 2023-09-30T01:00:52.103 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 bd06baef-4ba4-445d-918f-f168424bbc6a 2023-09-30T01:00:52.103 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:00:52.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEcxdldtP9BhAA+zPHigoyAiWLuKKDRY/YDg== 2023-09-30T01:00:52.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEcxdldtP9BhAA+zPHigoyAiWLuKKDRY/YDg=="}' 2023-09-30T01:00:52.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bd06baef-4ba4-445d-918f-f168424bbc6a -i td/mclock-config/0/new.json 2023-09-30T01:00:52.460 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:00:52.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2023-09-30T01:00:52.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBEcxdldtP9BhAA+zPHigoyAiWLuKKDRY/YDg== --osd-uuid bd06baef-4ba4-445d-918f-f168424bbc6a 2023-09-30T01:00:52.503 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:52.500+0000 7f29b9b2a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:52.506 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:52.500+0000 7f29b9b2a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:52.508 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:52.504+0000 7f29b9b2a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:52.509 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:52.504+0000 7f29b9b2a3c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:00:52.509 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:52.504+0000 7f29b9b2a3c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:00:52.509 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:52.504+0000 7f29b9b2a3c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:00:52.582 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:52.576+0000 7f29b9b2a3c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2023-09-30T01:00:58.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2023-09-30T01:00:58.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:00:58.760 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:00:58.760 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:00:58.760 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:00:59.260 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:00:59.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:00:59.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2023-09-30T01:00:59.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:00:59.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:00:59.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:00:59.292 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:59.288+0000 7f1ce6e903c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:59.294 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:59.288+0000 7f1ce6e903c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:59.295 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:59.292+0000 7f1ce6e903c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:00:59.838 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:00:59.838 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:00:59.838 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:00:59.838 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:00:59.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:00:59.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:00:59.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:00:59.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:00:59.839 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:00:59.832+0000 7f1ce6e903c0 -1 Falling back to public interface 2023-09-30T01:00:59.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:00:59.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:00.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:01.240 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:01.240 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:01.240 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:01:01.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:01:01.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:01.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:01.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:02.398 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:02.392+0000 7f1ce6e903c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:01:02.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:02.633 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:02.633 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:01:02.633 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:01:02.633 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:02.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:03.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:04.195 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:04.195 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:04.196 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:01:04.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:01:04.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:04.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:04.593 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:05.595 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:05.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:05.596 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:01:05.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:01:05.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:05.597 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:06.079 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:07.081 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:07.081 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:07.081 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:01:07.082 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:01:07.082 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:07.082 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:07.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:08.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:08.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:08.496 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:01:08.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:01:08.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:08.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:08.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:09.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:09.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:09.911 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:01:09.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:01:09.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:09.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:10.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:11.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:11.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:11.445 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:01:11.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:01:11.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:11.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:11.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:12.831 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:12.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:12.832 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:01:12.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:01:12.833 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:12.833 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:13.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:14.824 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:14.824 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:14.824 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:01:14.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:01:14.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:14.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:15.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:16.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:16.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:16.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:01:16.311 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:01:16.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:16.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:16.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:17.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:17.794 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:17.794 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:01:17.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:01:17.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:17.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:18.313 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:19.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:19.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:19.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:01:19.316 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:01:19.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:19.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:20.232 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:21.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:21.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:21.236 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:01:21.236 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:01:21.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:21.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:22.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:23.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:23.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:23.299 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:01:23.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:01:23.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:23.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:23.824 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:24.826 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:24.826 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:24.826 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:01:24.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:01:24.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:24.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:25.234 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34900,v1:127.0.0.1:6803/34900] [v2:127.0.0.1:6804/34900,v1:127.0.0.1:6805/34900] exists,up bd06baef-4ba4-445d-918f-f168424bbc6a 2023-09-30T01:01:25.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:01:25.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:01:25.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:01:25.235 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2023-09-30T01:01:25.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2023-09-30T01:01:25.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2023-09-30T01:01:25.670 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2023-09-30T01:01:25.670 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2023-09-30T01:01:25.670 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:01:25.671 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:01:25.671 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:01:25.671 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:25.671 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:25.672 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:01:25.672 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2023-09-30T01:01:25.672 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_profile 2023-09-30T01:01:25.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2023-09-30T01:01:25.767 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2023-09-30T01:01:25.767 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2023-09-30T01:01:25.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2023-09-30T01:01:25.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2023-09-30T01:01:25.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=("osd_mclock_scheduler_background_recovery_res" "osd_mclock_scheduler_client_res") 2023-09-30T01:01:25.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2023-09-30T01:01:25.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2023-09-30T01:01:25.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2023-09-30T01:01:25.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2023-09-30T01:01:25.769 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2023-09-30T01:01:25.769 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:01:25.769 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:01:25.769 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2023-09-30T01:01:25.769 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2023-09-30T01:01:25.769 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:01:25.769 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:25.770 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:25.770 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:01:25.770 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2023-09-30T01:01:25.770 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2023-09-30T01:01:25.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2023-09-30T01:01:25.853 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2023-09-30T01:01:25.853 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:25.856 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2023-09-30T01:01:25.856 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2023-09-30T01:01:26.443 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2023-09-30T01:01:26.444 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2023-09-30T01:01:26.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2023-09-30T01:01:26.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2023-09-30T01:01:26.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2023-09-30T01:01:28.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2023-09-30T01:01:28.581 INFO:tasks.workunit.client.0.smithi186.stdout:Check configuration values - Attempt#: 0 2023-09-30T01:01:28.581 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2023-09-30T01:01:28.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2023-09-30T01:01:28.999 INFO:tasks.workunit.client.0.smithi186.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2023-09-30T01:01:28.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2023-09-30T01:01:28.999 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2023-09-30T01:01:29.000 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:29.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2023-09-30T01:01:29.003 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2023-09-30T01:01:29.004 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2023-09-30T01:01:29.004 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2023-09-30T01:01:29.004 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2023-09-30T01:01:29.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2023-09-30T01:01:29.416 INFO:tasks.workunit.client.0.smithi186.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2023-09-30T01:01:29.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2023-09-30T01:01:29.417 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2023-09-30T01:01:29.417 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:29.421 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2023-09-30T01:01:29.421 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:29.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2023-09-30T01:01:29.423 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2023-09-30T01:01:29.423 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2023-09-30T01:01:29.424 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2023-09-30T01:01:29.424 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:01:29.425 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:01:29.426 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:01:29.426 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:29.426 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:29.427 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:01:29.427 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2023-09-30T01:01:29.427 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2023-09-30T01:01:29.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2023-09-30T01:01:29.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2023-09-30T01:01:29.512 INFO:tasks.workunit.client.0.smithi186.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2023-09-30T01:01:29.513 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2023-09-30T01:01:29.513 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:29.515 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2023-09-30T01:01:29.515 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:29.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2023-09-30T01:01:29.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2023-09-30T01:01:29.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2023-09-30T01:01:29.518 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2023-09-30T01:01:29.518 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2023-09-30T01:01:29.519 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2023-09-30T01:01:29.519 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2023-09-30T01:01:29.519 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:01:29.519 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:01:29.520 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:01:29.520 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:29.520 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:29.520 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:01:29.520 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2023-09-30T01:01:29.520 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2023-09-30T01:01:29.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2023-09-30T01:01:29.617 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2023-09-30T01:01:29.617 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:29.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2023-09-30T01:01:29.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2023-09-30T01:01:30.249 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2023-09-30T01:01:30.250 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2023-09-30T01:01:30.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2023-09-30T01:01:30.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2023-09-30T01:01:30.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2023-09-30T01:01:32.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2023-09-30T01:01:32.254 INFO:tasks.workunit.client.0.smithi186.stdout:Check configuration values - Attempt#: 0 2023-09-30T01:01:32.254 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2023-09-30T01:01:32.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2023-09-30T01:01:32.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2023-09-30T01:01:32.670 INFO:tasks.workunit.client.0.smithi186.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2023-09-30T01:01:32.671 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2023-09-30T01:01:32.671 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:32.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2023-09-30T01:01:32.673 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2023-09-30T01:01:32.674 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2023-09-30T01:01:32.674 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2023-09-30T01:01:32.674 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2023-09-30T01:01:33.055 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2023-09-30T01:01:33.056 INFO:tasks.workunit.client.0.smithi186.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2023-09-30T01:01:33.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2023-09-30T01:01:33.056 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2023-09-30T01:01:33.057 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:33.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2023-09-30T01:01:33.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:33.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2023-09-30T01:01:33.063 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2023-09-30T01:01:33.063 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2023-09-30T01:01:33.063 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2023-09-30T01:01:33.064 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:01:33.064 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:01:33.065 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:01:33.065 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:33.065 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:33.065 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:01:33.066 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2023-09-30T01:01:33.066 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2023-09-30T01:01:33.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2023-09-30T01:01:33.146 INFO:tasks.workunit.client.0.smithi186.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2023-09-30T01:01:33.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2023-09-30T01:01:33.146 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2023-09-30T01:01:33.147 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:33.149 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2023-09-30T01:01:33.149 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2023-09-30T01:01:33.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2023-09-30T01:01:33.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2023-09-30T01:01:33.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2023-09-30T01:01:33.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2023-09-30T01:01:33.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:01:33.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:01:33.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:01:33.152 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:01:33.153 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:01:33.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:01:33.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:01:33.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:01:33.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:01:33.263 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:01:33.264 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:01:33.265 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:01:33.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:01:33.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:01:33.268 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:01:33.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:01:33.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:01:33.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:01:33.271 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:01:33.273 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:01:33.275 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:01:33.275 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:01:33.287 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:01:33.287 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:33.287 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:33.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:01:33.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:01:33.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:01:33.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2023-09-30T01:01:33.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:01:33.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:01:33.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:01:33.289 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:01:33.289 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:01:33.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:01:33.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:01:33.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:01:33.294 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:01:33.294 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:01:33.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:01:33.296 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:01:33.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:01:33.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:01:33.299 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:01:33.301 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:01:33.301 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:01:33.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:01:33.302 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:01:33.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:01:33.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:01:33.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:01:33.307 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:01:33.307 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:33.307 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:33.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:01:33.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:01:33.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:01:33.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2023-09-30T01:01:33.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2023-09-30T01:01:33.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2023-09-30T01:01:33.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2023-09-30T01:01:33.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:01:33.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:01:33.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:01:33.312 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:01:33.312 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:01:33.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:01:33.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:01:33.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:01:33.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:01:33.315 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:01:33.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:01:33.317 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:01:33.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:01:33.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:01:33.319 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:01:33.321 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:01:33.321 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:01:33.321 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:01:33.321 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:01:33.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:01:33.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:01:33.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:01:33.327 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:01:33.327 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:33.328 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:33.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:01:33.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:01:33.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:01:33.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2023-09-30T01:01:33.331 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:01:33.331 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:33.331 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:33.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26283 2023-09-30T01:01:33.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:01:33.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:01:33.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:01:33.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2023-09-30T01:01:33.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2023-09-30T01:01:33.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2023-09-30T01:01:33.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2023-09-30T01:01:33.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2023-09-30T01:01:33.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2023-09-30T01:01:33.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:01:33.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:01:33.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:01:33.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:01:33.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:01:33.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:01:33.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:01:33.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:01:33.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:01:33.340 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:01:33.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:01:33.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:01:33.344 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:01:33.344 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:01:33.345 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:01:33.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:01:33.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:01:33.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:01:33.348 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:01:33.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:01:33.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:01:33.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:01:33.353 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:01:33.353 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:33.353 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:33.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:01:33.355 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:01:33.355 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:01:33.355 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2023-09-30T01:01:33.358 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:01:33.358 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:33.358 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:33.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26283 2023-09-30T01:01:33.361 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:01:33.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:01:33.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:01:33.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2023-09-30T01:01:33.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2023-09-30T01:01:33.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2023-09-30T01:01:33.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:01:33.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:01:33.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:01:33.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2023-09-30T01:01:33.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2023-09-30T01:01:33.730 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:01:33.731 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:01:33.731 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:01:33.731 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:01:33.731 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:33.732 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:33.732 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:01:33.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:01:34.029 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:01:34.031 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:01:34.031 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:01:34.031 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:01:34.031 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:01:34.031 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:01:34.032 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:01:34.032 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:01:34.032 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:01:34.033 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:01:34.033 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:34.033 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:34.033 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T01:01:34.034 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:01:34.034 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get fsid 2023-09-30T01:01:34.127 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:01:34.127 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:01:34.127 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:01:34.127 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:01:34.129 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:01:34.129 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:01:34.129 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:01:34.129 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:01:34.129 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:01:34.130 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:34.130 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:34.130 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T01:01:34.131 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:01:34.131 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get mon_host 2023-09-30T01:01:34.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2023-09-30T01:01:34.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2023-09-30T01:01:34.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:01:34.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:01:34.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:01:34.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2023-09-30T01:01:34.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:01:35.123 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:01:35.123 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:01:35.123 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:01:35.124 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:01:35.124 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:35.124 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:35.124 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:01:35.125 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:01:35.127 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:01:35.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2023-09-30T01:01:35.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2023-09-30T01:01:35.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:01:35.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:01:35.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:01:35.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2023-09-30T01:01:35.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2023-09-30T01:01:35.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:01:35.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:01:35.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:01:35.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2023-09-30T01:01:35.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2023-09-30T01:01:35.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:01:35.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:01:35.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2023-09-30T01:01:35.162 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:01:35.162 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:01:35.163 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:01:35.164 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:01:35.164 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:01:35.164 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:01:35.164 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:01:35.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:01:35.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:01:35.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:01:35.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:01:35.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2023-09-30T01:01:35.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2023-09-30T01:01:35.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:01:35.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:01:35.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:01:35.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:01:35.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:01:35.166 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2023-09-30T01:01:35.166 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2023-09-30T01:01:35.166 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:01:35.166 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f37e0b0f-ea08-4652-b316-72d2c72f270b 2023-09-30T01:01:35.167 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 f37e0b0f-ea08-4652-b316-72d2c72f270b 2023-09-30T01:01:35.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f37e0b0f-ea08-4652-b316-72d2c72f270b' 2023-09-30T01:01:35.167 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:01:35.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvcxdl/iLxChAAzNwy4MqYUm7W/dNRFu3WSA== 2023-09-30T01:01:35.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvcxdl/iLxChAAzNwy4MqYUm7W/dNRFu3WSA=="}' 2023-09-30T01:01:35.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f37e0b0f-ea08-4652-b316-72d2c72f270b -i td/mclock-config/0/new.json 2023-09-30T01:01:35.522 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:01:35.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2023-09-30T01:01:35.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBvcxdl/iLxChAAzNwy4MqYUm7W/dNRFu3WSA== --osd-uuid f37e0b0f-ea08-4652-b316-72d2c72f270b 2023-09-30T01:01:35.571 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:35.567+0000 7f6787e483c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:01:35.573 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:35.571+0000 7f6787e483c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:01:35.574 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:35.571+0000 7f6787e483c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:01:35.575 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:35.571+0000 7f6787e483c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:01:35.575 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:35.571+0000 7f6787e483c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:01:35.575 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:35.571+0000 7f6787e483c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:01:35.653 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:35.651+0000 7f6787e483c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2023-09-30T01:01:41.914 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2023-09-30T01:01:41.914 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:01:41.916 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:01:41.916 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:01:41.916 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:01:42.456 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:01:42.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:01:42.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2023-09-30T01:01:42.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:01:42.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:01:42.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:01:42.481 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:42.479+0000 7f41c50433c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:01:42.484 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:42.483+0000 7f41c50433c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:01:42.485 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:42.483+0000 7f41c50433c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:01:42.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:01:42.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:01:42.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:01:42.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:01:42.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:01:42.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:42.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:01:42.908 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:01:42.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:42.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:43.061 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:43.059+0000 7f41c50433c0 -1 Falling back to public interface 2023-09-30T01:01:43.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:44.279 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:44.279 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:44.279 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:01:44.280 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:01:44.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:44.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:44.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:45.605 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:01:45.603+0000 7f41c50433c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:01:45.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:45.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:45.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:01:45.685 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:01:45.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:45.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:46.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:47.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:47.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:47.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:01:47.284 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:01:47.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:47.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:47.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:48.700 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:48.701 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:48.701 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:01:48.701 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:01:48.701 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:48.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:49.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:50.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:50.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:50.115 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:01:50.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:01:50.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:50.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:50.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:51.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:51.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:51.542 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:01:51.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:01:51.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:51.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:51.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:52.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:52.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:52.973 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:01:52.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:01:52.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:52.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:53.379 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:54.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:54.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:54.381 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:01:54.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:01:54.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:54.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:54.759 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:55.761 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:55.761 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:55.761 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:01:55.761 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:01:55.762 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:55.762 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:56.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:57.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:57.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:57.210 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:01:57.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:01:57.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:57.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:57.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:01:58.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:01:58.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:01:58.997 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:01:58.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:01:58.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:01:58.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:01:59.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:00.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:00.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:00.685 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:02:00.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:02:00.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:00.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:01.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:02.195 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:02.195 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:02.195 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:02:02.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:02:02.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:02.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:03.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:04.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:04.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:04.211 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:02:04.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:02:04.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:04.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:05.075 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:06.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:06.078 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:06.078 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:02:06.078 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:02:06.078 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:06.079 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:06.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:07.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:07.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:07.779 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:02:07.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:02:07.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:07.780 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:08.233 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/37227,v1:127.0.0.1:6803/37227] [v2:127.0.0.1:6804/37227,v1:127.0.0.1:6805/37227] exists,up f37e0b0f-ea08-4652-b316-72d2c72f270b 2023-09-30T01:02:08.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:02:08.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:02:08.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:02:08.234 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2023-09-30T01:02:08.759 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2023-09-30T01:02:08.759 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2023-09-30T01:02:08.760 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2023-09-30T01:02:08.760 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2023-09-30T01:02:08.760 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:02:08.761 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:02:08.761 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:08.761 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:08.761 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:08.761 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:02:08.762 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2023-09-30T01:02:08.762 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_profile 2023-09-30T01:02:08.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2023-09-30T01:02:08.851 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2023-09-30T01:02:08.851 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2023-09-30T01:02:08.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2023-09-30T01:02:08.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2023-09-30T01:02:08.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=("osd_mclock_scheduler_background_recovery_res" "osd_mclock_scheduler_client_res") 2023-09-30T01:02:08.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2023-09-30T01:02:08.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2023-09-30T01:02:08.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2023-09-30T01:02:08.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2023-09-30T01:02:08.853 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2023-09-30T01:02:08.853 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2023-09-30T01:02:08.854 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:02:08.854 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:02:08.854 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2023-09-30T01:02:08.854 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:08.854 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:08.854 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:08.854 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:02:08.854 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2023-09-30T01:02:08.854 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2023-09-30T01:02:08.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2023-09-30T01:02:08.942 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2023-09-30T01:02:08.942 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:08.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2023-09-30T01:02:08.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2023-09-30T01:02:09.073 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:02:09.073 INFO:tasks.workunit.client.0.smithi186.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '0.800000' " 2023-09-30T01:02:09.073 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:02:09.084 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2023-09-30T01:02:09.085 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2023-09-30T01:02:09.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2023-09-30T01:02:09.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2023-09-30T01:02:09.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2023-09-30T01:02:11.089 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2023-09-30T01:02:11.089 INFO:tasks.workunit.client.0.smithi186.stdout:Check configuration values - Attempt#: 0 2023-09-30T01:02:11.090 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2023-09-30T01:02:11.499 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2023-09-30T01:02:11.499 INFO:tasks.workunit.client.0.smithi186.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2023-09-30T01:02:11.499 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2023-09-30T01:02:11.500 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2023-09-30T01:02:11.500 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:11.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2023-09-30T01:02:11.504 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2023-09-30T01:02:11.504 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2023-09-30T01:02:11.504 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2023-09-30T01:02:11.505 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2023-09-30T01:02:11.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2023-09-30T01:02:11.932 INFO:tasks.workunit.client.0.smithi186.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2023-09-30T01:02:11.933 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2023-09-30T01:02:11.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2023-09-30T01:02:11.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:11.937 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2023-09-30T01:02:11.937 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:11.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2023-09-30T01:02:11.942 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2023-09-30T01:02:11.942 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2023-09-30T01:02:11.943 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2023-09-30T01:02:11.943 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:02:11.943 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:02:11.943 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:11.943 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:11.943 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:11.943 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:02:11.943 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2023-09-30T01:02:11.943 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2023-09-30T01:02:12.022 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2023-09-30T01:02:12.023 INFO:tasks.workunit.client.0.smithi186.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2023-09-30T01:02:12.023 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2023-09-30T01:02:12.023 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2023-09-30T01:02:12.024 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:12.025 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2023-09-30T01:02:12.026 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:12.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2023-09-30T01:02:12.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2023-09-30T01:02:12.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2023-09-30T01:02:12.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2023-09-30T01:02:12.029 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2023-09-30T01:02:12.029 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2023-09-30T01:02:12.030 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2023-09-30T01:02:12.030 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:02:12.030 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:02:12.031 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:12.031 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:12.031 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:12.032 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:02:12.032 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2023-09-30T01:02:12.032 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2023-09-30T01:02:12.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2023-09-30T01:02:12.130 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:12.130 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2023-09-30T01:02:12.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2023-09-30T01:02:12.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2023-09-30T01:02:12.247 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:02:12.247 INFO:tasks.workunit.client.0.smithi186.stdout: "success": "osd_mclock_scheduler_client_res = '0.400000' " 2023-09-30T01:02:12.247 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:02:12.258 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2023-09-30T01:02:12.259 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2023-09-30T01:02:12.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2023-09-30T01:02:12.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2023-09-30T01:02:12.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2023-09-30T01:02:14.262 INFO:tasks.workunit.client.0.smithi186.stdout:Check configuration values - Attempt#: 0 2023-09-30T01:02:14.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2023-09-30T01:02:14.263 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2023-09-30T01:02:14.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2023-09-30T01:02:14.688 INFO:tasks.workunit.client.0.smithi186.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2023-09-30T01:02:14.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2023-09-30T01:02:14.689 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2023-09-30T01:02:14.689 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:14.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2023-09-30T01:02:14.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2023-09-30T01:02:14.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2023-09-30T01:02:14.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2023-09-30T01:02:14.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2023-09-30T01:02:15.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2023-09-30T01:02:15.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2023-09-30T01:02:15.119 INFO:tasks.workunit.client.0.smithi186.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2023-09-30T01:02:15.120 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2023-09-30T01:02:15.120 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:15.124 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2023-09-30T01:02:15.124 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:15.127 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2023-09-30T01:02:15.128 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2023-09-30T01:02:15.128 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2023-09-30T01:02:15.129 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2023-09-30T01:02:15.129 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:02:15.129 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:02:15.129 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:15.129 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:15.130 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:15.130 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:02:15.130 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2023-09-30T01:02:15.131 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2023-09-30T01:02:15.223 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2023-09-30T01:02:15.224 INFO:tasks.workunit.client.0.smithi186.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2023-09-30T01:02:15.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2023-09-30T01:02:15.224 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2023-09-30T01:02:15.224 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:15.227 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2023-09-30T01:02:15.228 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2023-09-30T01:02:15.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2023-09-30T01:02:15.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2023-09-30T01:02:15.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2023-09-30T01:02:15.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2023-09-30T01:02:15.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:02:15.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:02:15.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:02:15.232 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:02:15.232 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:02:15.232 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:02:15.232 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:02:15.232 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:02:15.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:02:15.344 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:02:15.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:02:15.346 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:02:15.348 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:02:15.348 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:02:15.349 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:02:15.351 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:15.351 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:02:15.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:02:15.352 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:15.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:02:15.355 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:02:15.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:02:15.365 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:02:15.365 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:15.365 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:15.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:02:15.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:02:15.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:02:15.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2023-09-30T01:02:15.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:02:15.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:02:15.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:02:15.368 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:02:15.368 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:02:15.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:02:15.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:02:15.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:02:15.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:02:15.372 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:02:15.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:02:15.374 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:02:15.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:02:15.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:02:15.375 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:02:15.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:15.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:02:15.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:02:15.378 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:15.380 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:02:15.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:02:15.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:02:15.383 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:02:15.383 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:15.384 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:15.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:02:15.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:02:15.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:02:15.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2023-09-30T01:02:15.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2023-09-30T01:02:15.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2023-09-30T01:02:15.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2023-09-30T01:02:15.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:02:15.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:02:15.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:02:15.388 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:02:15.388 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:02:15.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:02:15.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:02:15.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:02:15.390 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:02:15.391 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:02:15.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:02:15.392 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:02:15.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:02:15.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:02:15.395 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:02:15.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:15.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:02:15.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:02:15.397 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:15.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:02:15.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:02:15.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:02:15.403 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:02:15.403 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:15.403 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:15.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:02:15.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:02:15.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:02:15.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2023-09-30T01:02:15.408 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:02:15.408 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:15.408 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:15.408 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26283 2023-09-30T01:02:15.410 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:02:15.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:02:15.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:02:15.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2023-09-30T01:02:15.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2023-09-30T01:02:15.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2023-09-30T01:02:15.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2023-09-30T01:02:15.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2023-09-30T01:02:15.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2023-09-30T01:02:15.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:02:15.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:02:15.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:02:15.413 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:02:15.413 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:02:15.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:02:15.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:02:15.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:02:15.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:02:15.417 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:02:15.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:02:15.419 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:02:15.420 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:02:15.421 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:02:15.421 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:02:15.426 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:15.426 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:02:15.426 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:02:15.426 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:15.427 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:02:15.427 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:02:15.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:02:15.429 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:02:15.430 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:15.430 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:15.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:02:15.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:02:15.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:02:15.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2023-09-30T01:02:15.434 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:02:15.434 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:15.434 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:15.434 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26283 2023-09-30T01:02:15.436 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:02:15.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:02:15.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:02:15.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2023-09-30T01:02:15.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2023-09-30T01:02:15.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2023-09-30T01:02:15.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:02:15.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:02:15.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:02:15.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2023-09-30T01:02:15.439 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2023-09-30T01:02:15.833 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:02:15.834 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:02:15.834 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:02:15.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:02:15.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:15.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:15.835 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:02:15.838 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:02:16.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:02:16.116 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:02:16.116 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:02:16.116 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:02:16.116 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:02:16.116 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:02:16.117 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:02:16.117 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:02:16.117 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:02:16.118 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:16.118 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:16.118 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:16.118 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T01:02:16.119 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:02:16.119 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get fsid 2023-09-30T01:02:16.213 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:02:16.213 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:02:16.213 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:02:16.213 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:02:16.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:02:16.214 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:02:16.214 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:02:16.214 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:02:16.214 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:16.215 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:16.215 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:16.215 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-mon.a.asok 2023-09-30T01:02:16.215 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:02:16.215 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26283/ceph-mon.a.asok config get mon_host 2023-09-30T01:02:16.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2023-09-30T01:02:16.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2023-09-30T01:02:16.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:02:16.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:02:16.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:02:16.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2023-09-30T01:02:16.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:02:17.206 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:02:17.206 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:02:17.206 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:02:17.206 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:02:17.207 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:17.207 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:17.207 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:02:17.208 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:02:17.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:02:17.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2023-09-30T01:02:17.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2023-09-30T01:02:17.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:02:17.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:02:17.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:02:17.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2023-09-30T01:02:17.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2023-09-30T01:02:17.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:02:17.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:02:17.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:02:17.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2023-09-30T01:02:17.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2023-09-30T01:02:17.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:02:17.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:02:17.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2023-09-30T01:02:17.244 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:02:17.244 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:02:17.244 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:02:17.246 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:02:17.247 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:17.247 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:17.247 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:02:17.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' 2023-09-30T01:02:17.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:02:17.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:02:17.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:02:17.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2023-09-30T01:02:17.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2023-09-30T01:02:17.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:02:17.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:02:17.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:02:17.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:02:17.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:02:17.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2023-09-30T01:02:17.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2023-09-30T01:02:17.249 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:02:17.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=346079d3-470e-4130-a95c-188e0ee123c2 2023-09-30T01:02:17.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 346079d3-470e-4130-a95c-188e0ee123c2' 2023-09-30T01:02:17.251 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 346079d3-470e-4130-a95c-188e0ee123c2 2023-09-30T01:02:17.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:02:17.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZcxdloM7VDxAAmW+O4v66aR+ACozW4WdNgg== 2023-09-30T01:02:17.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZcxdloM7VDxAAmW+O4v66aR+ACozW4WdNgg=="}' 2023-09-30T01:02:17.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 346079d3-470e-4130-a95c-188e0ee123c2 -i td/mclock-config/0/new.json 2023-09-30T01:02:17.572 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:02:17.588 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2023-09-30T01:02:17.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCZcxdloM7VDxAAmW+O4v66aR+ACozW4WdNgg== --osd-uuid 346079d3-470e-4130-a95c-188e0ee123c2 2023-09-30T01:02:17.622 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:17.618+0000 7f64e39503c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:02:17.633 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:17.630+0000 7f64e39503c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:02:17.634 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:17.630+0000 7f64e39503c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:02:17.635 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:17.630+0000 7f64e39503c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:02:17.635 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:17.630+0000 7f64e39503c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:02:17.635 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:17.630+0000 7f64e39503c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2023-09-30T01:02:17.672 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:17.670+0000 7f64e39503c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2023-09-30T01:02:23.975 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2023-09-30T01:02:23.975 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:02:23.977 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:02:23.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:02:23.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:02:24.521 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:02:24.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:02:24.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e66ef939-9eaf-481e-b817-cbf7993f6ea1 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.26283/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2023-09-30T01:02:24.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:02:24.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:02:24.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:02:24.551 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:24.546+0000 7f277ef773c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:02:24.553 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:24.550+0000 7f277ef773c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:02:24.559 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:24.550+0000 7f277ef773c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:02:24.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:02:24.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:02:24.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:02:24.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:02:24.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:02:24.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:24.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:02:24.912 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:02:24.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:24.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:25.121 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:25.118+0000 7f277ef773c0 -1 Falling back to public interface 2023-09-30T01:02:25.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:26.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:26.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:26.329 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:02:26.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:02:26.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:26.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:26.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:27.625 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:27.622+0000 7f277ef773c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:02:27.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:27.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:27.712 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:02:27.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:02:27.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:27.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:28.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:29.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:29.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:29.226 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:02:29.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:02:29.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:29.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:29.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:30.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:30.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:30.673 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:02:30.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:02:30.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:30.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:31.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:32.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:32.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:32.095 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:02:32.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:02:32.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:32.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:32.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:33.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:33.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:33.509 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:02:33.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:02:33.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:33.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:33.945 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:34.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:34.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:34.947 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:02:34.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:02:34.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:34.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:35.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:36.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:36.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:36.445 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:02:36.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:02:36.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:36.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:36.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:37.896 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:37.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:37.897 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:02:37.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:02:37.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:37.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:38.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:39.348 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:39.348 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:39.348 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:02:39.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:02:39.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:39.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:40.207 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:41.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:41.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:41.210 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:02:41.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:02:41.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:41.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:42.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:43.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:43.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:43.243 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:02:43.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:02:43.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:43.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:44.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:45.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:45.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:45.325 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:02:45.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:02:45.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:45.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:46.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:47.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:47.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:47.098 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:02:47.099 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:02:47.099 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:47.099 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:47.896 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:02:48.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:02:48.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:02:48.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:02:48.898 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:02:48.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:02:48.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:02:49.387 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/39554,v1:127.0.0.1:6803/39554] [v2:127.0.0.1:6804/39554,v1:127.0.0.1:6805/39554] exists,up 346079d3-470e-4130-a95c-188e0ee123c2 2023-09-30T01:02:49.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:02:49.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:02:49.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:02:49.389 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T01:02:49.389 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:02:49.389 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:02:49.390 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:49.390 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:49.390 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:49.390 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:02:49.391 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T01:02:49.391 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_recovery_max_active 2023-09-30T01:02:49.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2023-09-30T01:02:49.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2023-09-30T01:02:49.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2023-09-30T01:02:49.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2023-09-30T01:02:50.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2023-09-30T01:02:52.181 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T01:02:52.181 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:02:52.181 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:02:52.182 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:52.182 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:52.182 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:52.182 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:02:52.183 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T01:02:52.183 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_recovery_max_active 2023-09-30T01:02:52.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2023-09-30T01:02:52.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2023-09-30T01:02:52.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2023-09-30T01:02:52.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2023-09-30T01:02:53.240 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2023-09-30T01:02:55.243 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2023-09-30T01:02:55.243 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:02:55.243 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:02:55.244 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:55.244 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:55.244 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:55.245 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26283/ceph-osd.0.asok 2023-09-30T01:02:55.245 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2023-09-30T01:02:55.245 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.26283/ceph-osd.0.asok config get osd_recovery_max_active 2023-09-30T01:02:55.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2023-09-30T01:02:55.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2023-09-30T01:02:55.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2023-09-30T01:02:55.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:02:55.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:02:55.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:02:55.344 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:02:55.344 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:02:55.344 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:02:55.344 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:02:55.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:02:55.458 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:02:55.458 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:02:55.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:02:55.461 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:02:55.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:02:55.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:02:55.464 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:02:55.465 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:55.465 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:02:55.466 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:02:55.466 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:55.468 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:02:55.470 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:02:55.471 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:02:55.479 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:02:55.479 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:55.479 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:55.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:02:55.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:02:55.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:02:55.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2023-09-30T01:02:55.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:02:55.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:02:55.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:02:55.483 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:02:55.483 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:02:55.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:02:55.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:02:55.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:02:55.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:02:55.488 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:02:55.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:02:55.490 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:02:55.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:02:55.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:02:55.493 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:02:55.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:55.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:02:55.495 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:55.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:02:55.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:02:55.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:02:55.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:02:55.500 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:02:55.500 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:55.500 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:55.500 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:02:55.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:02:55.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:02:55.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: main: code=0 2023-09-30T01:02:55.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: main: teardown td/mclock-config 0 2023-09-30T01:02:55.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2023-09-30T01:02:55.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-09-30T01:02:55.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2023-09-30T01:02:55.503 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:02:55.503 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:02:55.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:02:55.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:02:55.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:02:55.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:02:55.506 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:02:55.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:02:55.508 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:02:55.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:02:55.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:02:55.511 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:02:55.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:55.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:02:55.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:02:55.513 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:55.514 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:02:55.516 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-09-30T01:02:55.516 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2023-09-30T01:02:55.518 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:02:55.518 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:55.518 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26283 2023-09-30T01:02:55.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26283 2023-09-30T01:02:55.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:02:55.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:02:55.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: main: return 0 2023-09-30T01:02:55.521 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-30T01:02:55.521 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-30T01:02:55.580 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2023-09-30T01:02:55.580 DEBUG:teuthology.orchestra.run.smithi186:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=93dce5e17918f8aca65993a7a2dc40233a47b481 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2023-09-30T01:02:55.598 INFO:tasks.workunit.client.0.smithi186.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-09-30T01:02:55.612 INFO:tasks.workunit.client.0.smithi186.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-09-30T01:02:55.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-09-30T01:02:55.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-09-30T01:02:55.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: export PYTHONWARNINGS=ignore 2023-09-30T01:02:55.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: PYTHONWARNINGS=ignore 2023-09-30T01:02:55.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: export CEPH_CONF=/dev/null 2023-09-30T01:02:55.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: CEPH_CONF=/dev/null 2023-09-30T01:02:55.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: unset CEPH_ARGS 2023-09-30T01:02:55.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: main: local code 2023-09-30T01:02:55.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: main: run td/network-ping 2023-09-30T01:02:55.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2023-09-30T01:02:55.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2023-09-30T01:02:55.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2023-09-30T01:02:55.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2023-09-30T01:02:55.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2023-09-30T01:02:55.614 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2023-09-30T01:02:55.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none ' 2023-09-30T01:02:55.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2023-09-30T01:02:55.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2023-09-30T01:02:55.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2023-09-30T01:02:55.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2023-09-30T01:02:55.616 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2023-09-30T01:02:55.616 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-09-30T01:02:55.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2023-09-30T01:02:55.620 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_network_ping_test2' 2023-09-30T01:02:55.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2023-09-30T01:02:55.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2023-09-30T01:02:55.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2023-09-30T01:02:55.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2023-09-30T01:02:55.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2023-09-30T01:02:55.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:02:55.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2023-09-30T01:02:55.621 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:02:55.621 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:02:55.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:02:55.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:02:55.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:02:55.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:02:55.625 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:02:55.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:02:55.627 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:02:55.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:02:55.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:02:55.628 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:02:55.630 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:55.630 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:02:55.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:02:55.633 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:02:55.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:02:55.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:02:55.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2023-09-30T01:02:55.636 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:02:55.637 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:55.637 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:02:55.637 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.40580 2023-09-30T01:02:55.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:02:55.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:02:55.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2023-09-30T01:02:55.641 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:02:55.641 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:55.641 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:02:55.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.40580 2023-09-30T01:02:55.643 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:02:55.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-09-30T01:02:55.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-09-30T01:02:55.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:02:55.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2023-09-30T01:02:55.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2023-09-30T01:02:55.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2023-09-30T01:02:55.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2023-09-30T01:02:55.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2023-09-30T01:02:55.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:02:55.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:02:55.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:02:55.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2023-09-30T01:02:55.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2023-09-30T01:02:56.012 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:02:56.013 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:02:56.013 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:02:56.013 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:02:56.013 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:56.014 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:02:56.014 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:02:56.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:02:56.320 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:02:56.321 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:02:56.321 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:02:56.321 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:02:56.322 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:02:56.322 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:02:56.322 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:02:56.323 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:02:56.323 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:02:56.323 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:56.323 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:56.323 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:02:56.324 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-mon.a.asok 2023-09-30T01:02:56.324 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:02:56.324 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.40580/ceph-mon.a.asok config get fsid 2023-09-30T01:02:56.412 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:02:56.412 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:02:56.412 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:02:56.413 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:02:56.413 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:02:56.413 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:02:56.414 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:02:56.414 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:02:56.414 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:02:56.414 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:56.414 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:02:56.414 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-mon.a.asok 2023-09-30T01:02:56.414 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:02:56.415 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.40580/ceph-mon.a.asok config get mon_host 2023-09-30T01:02:56.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2023-09-30T01:02:56.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2023-09-30T01:02:56.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:02:56.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:02:56.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:02:56.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2023-09-30T01:02:56.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:02:57.408 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:02:57.408 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:02:57.408 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:02:57.409 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:02:57.409 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:57.409 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:02:57.409 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:02:57.410 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:02:57.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:02:57.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2023-09-30T01:02:57.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2023-09-30T01:02:57.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:02:57.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:02:57.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:02:57.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2023-09-30T01:02:57.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2023-09-30T01:02:57.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:02:57.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:02:57.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:02:57.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2023-09-30T01:02:57.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2023-09-30T01:02:57.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:02:57.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:02:57.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2023-09-30T01:02:57.483 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:02:57.483 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:02:57.483 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:02:57.484 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:02:57.484 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:02:57.484 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:02:57.485 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:02:57.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:02:57.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:02:57.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:02:57.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:02:57.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2023-09-30T01:02:57.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2023-09-30T01:02:57.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:02:57.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:02:57.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:02:57.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:02:57.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:02:57.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:02:57.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2023-09-30T01:02:57.487 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:02:57.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=734d048a-f4da-4c65-85b3-4fe765579964 2023-09-30T01:02:57.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 734d048a-f4da-4c65-85b3-4fe765579964' 2023-09-30T01:02:57.489 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 734d048a-f4da-4c65-85b3-4fe765579964 2023-09-30T01:02:57.489 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:02:57.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBcxdleWFjHhAAXKlELeVfhPESqr65bBUn/Q== 2023-09-30T01:02:57.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBcxdleWFjHhAAXKlELeVfhPESqr65bBUn/Q=="}' 2023-09-30T01:02:57.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 734d048a-f4da-4c65-85b3-4fe765579964 -i td/network-ping/0/new.json 2023-09-30T01:02:57.862 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:02:57.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2023-09-30T01:02:57.876 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBcxdleWFjHhAAXKlELeVfhPESqr65bBUn/Q== --osd-uuid 734d048a-f4da-4c65-85b3-4fe765579964 2023-09-30T01:02:57.909 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:57.905+0000 7f74e42da3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:02:57.911 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:57.909+0000 7f74e42da3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:02:57.913 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:57.909+0000 7f74e42da3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:02:57.914 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:57.909+0000 7f74e42da3c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2023-09-30T01:02:57.914 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:57.909+0000 7f74e42da3c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2023-09-30T01:02:57.914 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:57.909+0000 7f74e42da3c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2023-09-30T01:02:57.993 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:02:57.989+0000 7f74e42da3c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2023-09-30T01:03:04.361 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2023-09-30T01:03:04.361 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:03:04.363 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:03:04.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:03:04.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:03:04.906 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:03:04.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:03:04.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:03:04.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:03:04.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:03:04.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:03:04.938 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:04.933+0000 7fbf41eef3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:03:04.954 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:04.949+0000 7fbf41eef3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:03:04.955 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:04.953+0000 7fbf41eef3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:03:05.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:03:05.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:03:05.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:03:05.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:03:05.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:03:05.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:05.301 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:03:05.301 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:03:05.301 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:05.301 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:05.502 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:05.497+0000 7fbf41eef3c0 -1 Falling back to public interface 2023-09-30T01:03:05.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:06.773 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:06.773 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:06.773 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:03:06.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:03:06.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:06.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:07.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:08.026 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:08.021+0000 7fbf41eef3c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:03:08.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:08.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:08.166 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:03:08.166 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:03:08.166 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:08.166 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:08.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:09.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:09.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:09.620 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:03:09.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:03:09.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:09.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:10.071 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:11.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:11.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:11.074 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:03:11.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:03:11.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:11.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:11.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:12.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:12.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:12.485 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:03:12.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:03:12.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:12.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:12.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:13.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:13.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:14.000 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:03:14.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:03:14.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:14.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:14.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:15.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:15.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:15.404 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:03:15.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:03:15.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:15.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:15.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:16.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:16.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:16.829 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:03:16.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:03:16.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:16.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:17.236 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:18.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:18.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:18.239 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:03:18.240 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:03:18.240 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:18.240 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:19.111 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:20.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:20.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:20.113 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:03:20.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:03:20.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:20.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:21.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:22.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:22.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:22.153 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:03:22.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:03:22.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:22.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:23.081 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:24.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:24.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:24.083 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:03:24.084 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:03:24.084 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:24.084 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:25.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:26.051 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:26.052 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:26.052 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:03:26.052 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:03:26.052 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:26.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:26.873 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:27.874 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:27.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:27.875 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:03:27.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:03:27.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:27.876 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:28.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:29.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:29.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:29.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:03:29.992 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:03:29.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:29.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:03:30.445 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/41710,v1:127.0.0.1:6803/41710] [v2:127.0.0.1:6804/41710,v1:127.0.0.1:6805/41710] exists,up 734d048a-f4da-4c65-85b3-4fe765579964 2023-09-30T01:03:30.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:03:30.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:03:30.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:03:30.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2023-09-30T01:03:30.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2023-09-30T01:03:30.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:03:30.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-09-30T01:03:30.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:03:30.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2023-09-30T01:03:30.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2023-09-30T01:03:30.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:03:30.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:03:30.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:03:30.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2023-09-30T01:03:30.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2023-09-30T01:03:30.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:03:30.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:03:30.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2023-09-30T01:03:30.448 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:03:30.448 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:03:30.448 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:03:30.450 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:03:30.450 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:03:30.450 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:03:30.450 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:03:30.450 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:03:30.450 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:03:30.450 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:03:30.450 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:03:30.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2023-09-30T01:03:30.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2023-09-30T01:03:30.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:03:30.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:03:30.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:03:30.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:03:30.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:03:30.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:03:30.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2023-09-30T01:03:30.452 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:03:30.452 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4de5027a-fe08-4a1e-8c43-ede98feaf75a 2023-09-30T01:03:30.452 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4de5027a-fe08-4a1e-8c43-ede98feaf75a' 2023-09-30T01:03:30.452 INFO:tasks.workunit.client.0.smithi186.stdout:add osd1 4de5027a-fe08-4a1e-8c43-ede98feaf75a 2023-09-30T01:03:30.453 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:03:30.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDicxdlb3JBHBAAfOynA6bsF8yQQZtYAkupMQ== 2023-09-30T01:03:30.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDicxdlb3JBHBAAfOynA6bsF8yQQZtYAkupMQ=="}' 2023-09-30T01:03:30.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4de5027a-fe08-4a1e-8c43-ede98feaf75a -i td/network-ping/1/new.json 2023-09-30T01:03:31.041 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:03:31.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2023-09-30T01:03:31.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDicxdlb3JBHBAAfOynA6bsF8yQQZtYAkupMQ== --osd-uuid 4de5027a-fe08-4a1e-8c43-ede98feaf75a 2023-09-30T01:03:31.096 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:31.092+0000 7f389645a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:03:31.122 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:31.116+0000 7f389645a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:03:31.124 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:31.120+0000 7f389645a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:03:31.125 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:31.120+0000 7f389645a3c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2023-09-30T01:03:31.125 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:31.120+0000 7f389645a3c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2023-09-30T01:03:31.125 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:31.120+0000 7f389645a3c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2023-09-30T01:03:31.205 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:31.200+0000 7f389645a3c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2023-09-30T01:03:37.089 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2023-09-30T01:03:37.089 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:03:37.091 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd1 key to auth repository 2023-09-30T01:03:37.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-09-30T01:03:37.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:03:37.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-09-30T01:03:37.625 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:03:37.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:03:37.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:03:37.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:03:37.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:03:37.649 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:37.644+0000 7fd050e1d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:03:37.652 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:37.648+0000 7fd050e1d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:03:37.658 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:37.648+0000 7fd050e1d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:03:38.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-09-30T01:03:38.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:03:38.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:03:38.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:03:38.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:03:38.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:38.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:03:38.061 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:03:38.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:38.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:38.198 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:38.192+0000 7fd050e1d3c0 -1 Falling back to public interface 2023-09-30T01:03:38.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:39.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:39.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:39.542 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:03:39.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:03:39.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:39.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:39.945 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:40.594 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:03:40.588+0000 7fd050e1d3c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:03:40.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:40.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:40.947 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:03:40.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:03:40.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:40.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:41.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:42.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:42.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:42.416 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:03:42.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:03:42.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:42.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:42.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:43.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:43.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:43.843 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:03:43.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:03:43.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:43.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:44.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:45.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:45.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:45.396 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:03:45.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:03:45.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:45.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:45.943 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:46.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:46.945 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:46.945 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:03:46.945 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:03:46.945 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:46.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:47.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:48.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:48.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:48.520 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:03:48.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:03:48.521 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:48.521 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:48.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:49.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:49.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:49.967 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:03:49.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:03:49.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:49.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:50.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:51.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:51.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:51.496 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:03:51.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:03:51.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:51.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:51.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:52.936 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:52.936 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:52.937 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:03:52.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:03:52.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:52.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:53.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:54.904 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:54.904 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:54.905 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:03:54.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:03:54.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:54.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:55.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:56.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:56.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:56.985 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:03:56.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:03:56.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:56.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:57.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:03:58.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:03:58.981 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:03:58.981 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:03:58.981 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:03:58.981 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:03:58.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:03:59.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:00.601 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:00.601 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:00.602 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:04:00.602 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:04:00.602 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:00.602 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:04:01.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:02.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:02.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:02.561 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:04:02.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:04:02.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:02.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:04:03.067 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:04.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:04.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:04.070 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:04:04.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:04:04.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:04.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:04:04.481 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/43433,v1:127.0.0.1:6811/43433] [v2:127.0.0.1:6812/43433,v1:127.0.0.1:6813/43433] exists,up 4de5027a-fe08-4a1e-8c43-ede98feaf75a 2023-09-30T01:04:04.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:04:04.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:04:04.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:04:04.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2023-09-30T01:04:04.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2023-09-30T01:04:04.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:04:04.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-09-30T01:04:04.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:04:04.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2023-09-30T01:04:04.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2023-09-30T01:04:04.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:04:04.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:04:04.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:04:04.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2023-09-30T01:04:04.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2023-09-30T01:04:04.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:04:04.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:04:04.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2023-09-30T01:04:04.485 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:04:04.485 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:04:04.485 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:04:04.486 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:04:04.486 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:04:04.487 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:04:04.487 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:04:04.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:04:04.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:04:04.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:04:04.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:04:04.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2023-09-30T01:04:04.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2023-09-30T01:04:04.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:04:04.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:04:04.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:04:04.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:04:04.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:04:04.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:04:04.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2023-09-30T01:04:04.488 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:04:04.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c5d6ff1d-0d4e-4661-8eda-eb4a4ecf5ce2 2023-09-30T01:04:04.489 INFO:tasks.workunit.client.0.smithi186.stdout:add osd2 c5d6ff1d-0d4e-4661-8eda-eb4a4ecf5ce2 2023-09-30T01:04:04.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c5d6ff1d-0d4e-4661-8eda-eb4a4ecf5ce2' 2023-09-30T01:04:04.489 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:04:04.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAEdBdl2q41HhAA9hghXSb5wQ25Jg6Ln7K/Gg== 2023-09-30T01:04:04.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAEdBdl2q41HhAA9hghXSb5wQ25Jg6Ln7K/Gg=="}' 2023-09-30T01:04:04.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c5d6ff1d-0d4e-4661-8eda-eb4a4ecf5ce2 -i td/network-ping/2/new.json 2023-09-30T01:04:05.011 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:04:05.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2023-09-30T01:04:05.027 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEdBdl2q41HhAA9hghXSb5wQ25Jg6Ln7K/Gg== --osd-uuid c5d6ff1d-0d4e-4661-8eda-eb4a4ecf5ce2 2023-09-30T01:04:05.062 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:05.056+0000 7fd19faa33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:04:05.065 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:05.060+0000 7fd19faa33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:04:05.067 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:05.060+0000 7fd19faa33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:04:05.068 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:05.064+0000 7fd19faa33c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2023-09-30T01:04:05.069 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:05.064+0000 7fd19faa33c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2023-09-30T01:04:05.069 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:05.064+0000 7fd19faa33c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2023-09-30T01:04:05.166 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:05.160+0000 7fd19faa33c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2023-09-30T01:04:11.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2023-09-30T01:04:11.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:04:11.416 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd2 key to auth repository 2023-09-30T01:04:11.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-09-30T01:04:11.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:04:11.948 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.2 2023-09-30T01:04:11.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-09-30T01:04:11.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:04:11.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:04:11.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:04:11.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:04:11.981 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:11.980+0000 7fc63db133c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:04:11.984 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:11.984+0000 7fc63db133c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:04:11.988 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:11.984+0000 7fc63db133c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:04:12.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-09-30T01:04:12.393 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:04:12.393 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-09-30T01:04:12.393 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:04:12.393 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:04:12.393 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:12.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:04:12.394 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:04:12.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:12.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:12.546 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:12.544+0000 7fc63db133c0 -1 Falling back to public interface 2023-09-30T01:04:12.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:13.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:13.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:13.827 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:04:13.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:04:13.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:13.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:14.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:14.763 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:04:14.759+0000 7fc63db133c0 -1 osd.2 0 log_to_monitors true 2023-09-30T01:04:15.236 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:15.236 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:15.236 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:04:15.236 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:04:15.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:15.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:15.864 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:16.865 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:16.866 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:16.866 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:04:16.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:04:16.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:16.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:17.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:18.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:18.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:18.328 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:04:18.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:04:18.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:18.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:18.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:19.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:19.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:19.830 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:04:19.831 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:04:19.831 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:19.831 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:20.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:21.271 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:21.271 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:21.271 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:04:21.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:04:21.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:21.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:21.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:22.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:22.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:22.699 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:04:22.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:04:22.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:22.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:23.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:24.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:24.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:24.118 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:04:24.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:04:24.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:24.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:24.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:25.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:25.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:25.669 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:04:25.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:04:25.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:25.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:26.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:27.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:27.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:27.350 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:04:27.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:04:27.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:27.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:27.925 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:28.927 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:28.928 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:28.928 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:04:28.928 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:04:28.928 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:28.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:29.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:30.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:30.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:30.804 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:04:30.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:04:30.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:30.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:31.592 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:32.594 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:32.594 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:32.595 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:04:32.595 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:04:32.595 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:32.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:33.240 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:34.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:34.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:34.242 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:04:34.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:04:34.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:34.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:35.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:36.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:36.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:36.337 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:04:36.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:04:36.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:36.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:36.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:04:37.834 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:04:37.834 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:04:37.835 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:04:37.836 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:04:37.836 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:04:37.836 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:04:38.245 INFO:tasks.workunit.client.0.smithi186.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/45182,v1:127.0.0.1:6819/45182] [v2:127.0.0.1:6820/45182,v1:127.0.0.1:6821/45182] exists,up c5d6ff1d-0d4e-4661-8eda-eb4a4ecf5ce2 2023-09-30T01:04:38.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:04:38.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:04:38.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:04:38.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2023-09-30T01:04:43.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2023-09-30T01:04:43.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2023-09-30T01:04:44.165 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'foo' already exists 2023-09-30T01:04:44.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:04:45.179 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2023-09-30T01:04:45.324 INFO:tasks.workunit.client.0.smithi186.stdout:hints = 1 2023-09-30T01:04:45.324 INFO:tasks.workunit.client.0.smithi186.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2023-09-30T01:04:45.324 INFO:tasks.workunit.client.0.smithi186.stdout:Object prefix: benchmark_data_smithi186_46125 2023-09-30T01:04:45.324 INFO:tasks.workunit.client.0.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-09-30T01:04:45.324 INFO:tasks.workunit.client.0.smithi186.stdout: 0 6 6 0 0 0 - 0 2023-09-30T01:04:46.324 INFO:tasks.workunit.client.0.smithi186.stdout: 1 16 64 48 0.187427 0.1875 0.423592 0.307429 2023-09-30T01:04:47.324 INFO:tasks.workunit.client.0.smithi186.stdout: 2 16 128 112 0.218691 0.25 0.217317 0.270964 2023-09-30T01:04:48.324 INFO:tasks.workunit.client.0.smithi186.stdout: 3 16 192 176 0.229117 0.25 0.248583 0.266531 2023-09-30T01:04:49.324 INFO:tasks.workunit.client.0.smithi186.stdout: 4 16 242 226 0.220659 0.195312 0.240037 0.271534 2023-09-30T01:04:50.324 INFO:tasks.workunit.client.0.smithi186.stdout: 5 16 306 290 0.22652 0.25 0.224111 0.269718 2023-09-30T01:04:51.325 INFO:tasks.workunit.client.0.smithi186.stdout: 6 16 368 352 0.229124 0.242188 0.230152 0.265601 2023-09-30T01:04:52.325 INFO:tasks.workunit.client.0.smithi186.stdout: 7 16 432 416 0.232101 0.25 0.232627 0.265181 2023-09-30T01:04:53.325 INFO:tasks.workunit.client.0.smithi186.stdout: 8 16 496 480 0.234333 0.25 0.293058 0.264334 2023-09-30T01:04:54.325 INFO:tasks.workunit.client.0.smithi186.stdout: 9 16 560 544 0.236069 0.25 0.258886 0.26357 2023-09-30T01:04:55.325 INFO:tasks.workunit.client.0.smithi186.stdout: 10 16 610 594 0.231992 0.195312 0.293666 0.263924 2023-09-30T01:04:56.325 INFO:tasks.workunit.client.0.smithi186.stdout:Total time run: 10.1779 2023-09-30T01:04:56.326 INFO:tasks.workunit.client.0.smithi186.stdout:Total writes made: 610 2023-09-30T01:04:56.326 INFO:tasks.workunit.client.0.smithi186.stdout:Write size: 4096 2023-09-30T01:04:56.326 INFO:tasks.workunit.client.0.smithi186.stdout:Object size: 4096 2023-09-30T01:04:56.326 INFO:tasks.workunit.client.0.smithi186.stdout:Bandwidth (MB/sec): 0.234116 2023-09-30T01:04:56.326 INFO:tasks.workunit.client.0.smithi186.stdout:Stddev Bandwidth: 0.0273251 2023-09-30T01:04:56.326 INFO:tasks.workunit.client.0.smithi186.stdout:Max bandwidth (MB/sec): 0.25 2023-09-30T01:04:56.326 INFO:tasks.workunit.client.0.smithi186.stdout:Min bandwidth (MB/sec): 0.1875 2023-09-30T01:04:56.326 INFO:tasks.workunit.client.0.smithi186.stdout:Average IOPS: 59 2023-09-30T01:04:56.326 INFO:tasks.workunit.client.0.smithi186.stdout:Stddev IOPS: 6.99524 2023-09-30T01:04:56.327 INFO:tasks.workunit.client.0.smithi186.stdout:Max IOPS: 64 2023-09-30T01:04:56.327 INFO:tasks.workunit.client.0.smithi186.stdout:Min IOPS: 48 2023-09-30T01:04:56.327 INFO:tasks.workunit.client.0.smithi186.stdout:Average Latency(s): 0.264412 2023-09-30T01:04:56.327 INFO:tasks.workunit.client.0.smithi186.stdout:Stddev Latency(s): 0.039053 2023-09-30T01:04:56.327 INFO:tasks.workunit.client.0.smithi186.stdout:Max latency(s): 0.423728 2023-09-30T01:04:56.327 INFO:tasks.workunit.client.0.smithi186.stdout:Min latency(s): 0.207093 2023-09-30T01:04:56.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2023-09-30T01:05:06.440 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2023-09-30T01:05:06.441 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:05:06.441 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:05:06.846 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2023-09-30T01:05:06.846 INFO:tasks.workunit.client.0.smithi186.stderr:1 2023-09-30T01:05:06.847 INFO:tasks.workunit.client.0.smithi186.stderr:2' 2023-09-30T01:05:06.847 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:05:06.847 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:05:06.847 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:05:07.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836511 2023-09-30T01:05:07.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836511 2023-09-30T01:05:07.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836511' 2023-09-30T01:05:07.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:05:07.013 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-09-30T01:05:07.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672984 2023-09-30T01:05:07.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672984 2023-09-30T01:05:07.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836511 1-42949672984' 2023-09-30T01:05:07.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:05:07.164 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-09-30T01:05:07.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509456 2023-09-30T01:05:07.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509456 2023-09-30T01:05:07.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836511 1-42949672984 2-64424509456' 2023-09-30T01:05:07.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:05:07.309 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836511 2023-09-30T01:05:07.309 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:05:07.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:05:07.312 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836511 2023-09-30T01:05:07.312 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:05:07.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836511 2023-09-30T01:05:07.314 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836511 2023-09-30T01:05:07.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2023-09-30T01:05:07.315 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:05:07.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836509 -lt 21474836511 2023-09-30T01:05:07.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:05:08.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:05:08.711 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:05:09.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836511 -lt 21474836511 2023-09-30T01:05:09.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:05:09.147 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672984 2023-09-30T01:05:09.147 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:05:09.149 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2023-09-30T01:05:09.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672984 2023-09-30T01:05:09.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:05:09.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672984 2023-09-30T01:05:09.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2023-09-30T01:05:09.152 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.1 seq 42949672984 2023-09-30T01:05:09.153 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:05:09.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672984 -lt 42949672984 2023-09-30T01:05:09.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:05:09.531 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509456 2023-09-30T01:05:09.532 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:05:09.534 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2023-09-30T01:05:09.535 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509456 2023-09-30T01:05:09.535 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:05:09.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509456 2023-09-30T01:05:09.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2023-09-30T01:05:09.537 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.2 seq 64424509456 2023-09-30T01:05:09.538 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2023-09-30T01:05:10.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509457 -lt 64424509456 2023-09-30T01:05:10.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:05:10.010 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2023-09-30T01:05:10.011 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:05:10.011 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:05:10.011 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:05:10.011 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:05:10.012 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:05:10.012 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-osd.0.asok 2023-09-30T01:05:10.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:05:10.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-osd.0.asok dump_osd_network 2023-09-30T01:05:10.098 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:05:10.099 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 1000, 2023-09-30T01:05:10.099 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [] 2023-09-30T01:05:10.099 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:05:10.107 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:10.108 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:05:10.149 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:05:10.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:10.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2023-09-30T01:05:10.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2023-09-30T01:05:10.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:05:10.192 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2023-09-30T01:05:10.192 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2023-09-30T01:05:10.192 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2023-09-30T01:05:10.192 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:05:10.192 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:05:10.192 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:05:10.192 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-mgr.x.asok 2023-09-30T01:05:10.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:05:10.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-mgr.x.asok dump_osd_network 2023-09-30T01:05:10.268 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:05:10.268 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 1000, 2023-09-30T01:05:10.268 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [] 2023-09-30T01:05:10.268 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:05:10.276 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:10.277 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:05:10.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:05:10.319 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:10.319 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2023-09-30T01:05:10.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2023-09-30T01:05:10.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:05:10.360 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2023-09-30T01:05:10.361 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:05:10.361 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:05:10.361 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:05:10.361 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:05:10.362 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:05:10.362 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-osd.0.asok 2023-09-30T01:05:10.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:05:10.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-osd.0.asok dump_osd_network 0 2023-09-30T01:05:10.448 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:05:10.448 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 0, 2023-09-30T01:05:10.448 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [ 2023-09-30T01:05:10.449 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.449 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:09 2023", 2023-09-30T01:05:10.449 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.449 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:10.449 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:10.449 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:10.449 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.449 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.563, 2023-09-30T01:05:10.449 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.540, 2023-09-30T01:05:10.450 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.585 2023-09-30T01:05:10.450 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.450 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.450 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.653, 2023-09-30T01:05:10.450 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.653, 2023-09-30T01:05:10.450 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.724 2023-09-30T01:05:10.450 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.450 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.451 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.653, 2023-09-30T01:05:10.451 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.653, 2023-09-30T01:05:10.451 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.724 2023-09-30T01:05:10.451 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.451 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.625 2023-09-30T01:05:10.451 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.451 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.451 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:09 2023", 2023-09-30T01:05:10.451 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.451 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:10.452 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:10.452 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:10.452 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.452 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.516, 2023-09-30T01:05:10.452 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.532, 2023-09-30T01:05:10.452 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.552 2023-09-30T01:05:10.452 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.452 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.452 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.566, 2023-09-30T01:05:10.453 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.677, 2023-09-30T01:05:10.453 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.677 2023-09-30T01:05:10.453 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.453 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.453 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.566, 2023-09-30T01:05:10.453 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.677, 2023-09-30T01:05:10.453 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.677 2023-09-30T01:05:10.453 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.453 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.523 2023-09-30T01:05:10.454 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.454 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.454 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:09 2023", 2023-09-30T01:05:10.454 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.454 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:10.454 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:10.454 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:10.454 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.454 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.549, 2023-09-30T01:05:10.455 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.532, 2023-09-30T01:05:10.455 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.512 2023-09-30T01:05:10.455 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.455 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.455 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.631, 2023-09-30T01:05:10.455 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.793, 2023-09-30T01:05:10.455 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.793 2023-09-30T01:05:10.455 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.455 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.456 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.631, 2023-09-30T01:05:10.456 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.793, 2023-09-30T01:05:10.456 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.793 2023-09-30T01:05:10.456 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.456 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.489 2023-09-30T01:05:10.456 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.456 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.456 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:09 2023", 2023-09-30T01:05:10.456 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.456 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:10.457 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:10.457 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:10.457 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.457 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.464, 2023-09-30T01:05:10.457 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.458, 2023-09-30T01:05:10.457 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.481 2023-09-30T01:05:10.457 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.457 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.457 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.579, 2023-09-30T01:05:10.458 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.651, 2023-09-30T01:05:10.458 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.651 2023-09-30T01:05:10.458 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.458 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.458 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.579, 2023-09-30T01:05:10.458 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.651, 2023-09-30T01:05:10.458 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.651 2023-09-30T01:05:10.458 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.458 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.448 2023-09-30T01:05:10.458 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:05:10.459 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:05:10.459 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:05:10.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:05:10.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:10.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2023-09-30T01:05:10.495 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:10.495 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2023-09-30T01:05:10.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:05:10.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:05:10.536 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2023-09-30T01:05:10.537 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2023-09-30T01:05:10.537 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2023-09-30T01:05:10.538 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:05:10.538 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:05:10.538 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:05:10.538 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-mgr.x.asok 2023-09-30T01:05:10.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:05:10.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-mgr.x.asok dump_osd_network 0 2023-09-30T01:05:10.612 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:05:10.612 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 0, 2023-09-30T01:05:10.612 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [ 2023-09-30T01:05:10.612 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.612 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:08 2023", 2023-09-30T01:05:10.612 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.612 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:05:10.612 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:10.612 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:10.612 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.613 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.580, 2023-09-30T01:05:10.613 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.598, 2023-09-30T01:05:10.613 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.592 2023-09-30T01:05:10.613 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.613 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.613 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.475, 2023-09-30T01:05:10.613 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.389, 2023-09-30T01:05:10.613 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.389 2023-09-30T01:05:10.613 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.613 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.613 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.654, 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.727, 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.727 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.602 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:09 2023", 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:10.614 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.563, 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.540, 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.585 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.469, 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.367, 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.367 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.653, 2023-09-30T01:05:10.615 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.653, 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.724 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.625 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:07 2023", 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.616 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.573, 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.568, 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.581 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.504, 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.442, 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.442 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.651, 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.713, 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.713 2023-09-30T01:05:10.617 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.600 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:07 2023", 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.564, 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.559, 2023-09-30T01:05:10.618 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.572 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.518, 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.386, 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.386 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.611, 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.642, 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.642 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.619 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.570 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:08 2023", 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.563, 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.548, 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.518 2023-09-30T01:05:10.620 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.527, 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.360, 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.279 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.653, 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.735, 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.756 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.531 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.621 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.622 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:08 2023", 2023-09-30T01:05:10.622 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.622 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:05:10.622 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:10.622 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:10.622 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.622 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.527, 2023-09-30T01:05:10.622 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.554, 2023-09-30T01:05:10.622 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.557 2023-09-30T01:05:10.622 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.622 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.392, 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.392, 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.392 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.640, 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.707, 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.712 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.640 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.623 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:09 2023", 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.516, 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.532, 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.552 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.624 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.402, 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.264, 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.264 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.566, 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.677, 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.677 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.523 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.625 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:09 2023", 2023-09-30T01:05:10.626 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.626 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:10.626 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:10.626 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:10.626 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.626 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.549, 2023-09-30T01:05:10.626 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.532, 2023-09-30T01:05:10.626 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.512 2023-09-30T01:05:10.626 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.626 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.626 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.471, 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.344, 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.306 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.631, 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.793, 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.793 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.489 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:07 2023", 2023-09-30T01:05:10.627 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.506, 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.517, 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.529 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.431, 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.431, 2023-09-30T01:05:10.628 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.431 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.549, 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.607, 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.616 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.431 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:08 2023", 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.629 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:05:10.630 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:05:10.630 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:10.630 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.630 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.513, 2023-09-30T01:05:10.630 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.512, 2023-09-30T01:05:10.630 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.485 2023-09-30T01:05:10.630 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.630 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.630 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.453, 2023-09-30T01:05:10.630 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.441, 2023-09-30T01:05:10.630 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.285 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.607, 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.611, 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.645 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.566 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:07 2023", 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:05:10.631 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.490, 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.487, 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.487 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.432, 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.369, 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.369 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.632 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.616, 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.690, 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.690 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.505 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:09 2023", 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:10.633 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:10.634 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:10.634 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.464, 2023-09-30T01:05:10.634 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.458, 2023-09-30T01:05:10.634 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.481 2023-09-30T01:05:10.634 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.634 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:10.634 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.312, 2023-09-30T01:05:10.634 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.291, 2023-09-30T01:05:10.634 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.291 2023-09-30T01:05:10.634 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.634 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:10.635 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.579, 2023-09-30T01:05:10.635 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.651, 2023-09-30T01:05:10.635 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.651 2023-09-30T01:05:10.635 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:10.635 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.448 2023-09-30T01:05:10.635 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:05:10.635 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:05:10.635 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:05:10.636 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:10.636 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:05:10.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2023-09-30T01:05:10.662 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:10.662 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2023-09-30T01:05:10.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:05:10.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2023-09-30T01:05:30.705 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2023-09-30T01:05:30.706 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:05:30.706 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:05:31.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2023-09-30T01:05:31.116 INFO:tasks.workunit.client.0.smithi186.stderr:1 2023-09-30T01:05:31.116 INFO:tasks.workunit.client.0.smithi186.stderr:2' 2023-09-30T01:05:31.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:05:31.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:05:31.116 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:05:31.285 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836518 2023-09-30T01:05:31.285 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836518 2023-09-30T01:05:31.285 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836518' 2023-09-30T01:05:31.285 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:05:31.286 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-09-30T01:05:31.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672991 2023-09-30T01:05:31.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672991 2023-09-30T01:05:31.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836518 1-42949672991' 2023-09-30T01:05:31.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:05:31.429 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-09-30T01:05:31.575 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509463 2023-09-30T01:05:31.575 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509463 2023-09-30T01:05:31.575 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836518 1-42949672991 2-64424509463' 2023-09-30T01:05:31.575 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:05:31.576 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836518 2023-09-30T01:05:31.576 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:05:31.576 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:05:31.577 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836518 2023-09-30T01:05:31.577 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:05:31.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836518 2023-09-30T01:05:31.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2023-09-30T01:05:31.580 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836518 2023-09-30T01:05:31.580 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:05:32.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836516 -lt 21474836518 2023-09-30T01:05:32.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:05:33.050 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:05:33.051 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:05:33.458 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836518 -lt 21474836518 2023-09-30T01:05:33.458 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:05:33.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672991 2023-09-30T01:05:33.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:05:33.461 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2023-09-30T01:05:33.463 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672991 2023-09-30T01:05:33.463 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:05:33.464 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672991 2023-09-30T01:05:33.465 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2023-09-30T01:05:33.465 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.1 seq 42949672991 2023-09-30T01:05:33.466 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:05:33.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672991 -lt 42949672991 2023-09-30T01:05:33.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:05:33.888 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509463 2023-09-30T01:05:33.888 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:05:33.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2023-09-30T01:05:33.891 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509463 2023-09-30T01:05:33.891 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:05:33.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509463 2023-09-30T01:05:33.894 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.2 seq 64424509463 2023-09-30T01:05:33.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2023-09-30T01:05:33.895 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2023-09-30T01:05:34.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509463 -lt 64424509463 2023-09-30T01:05:34.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:05:34.348 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2023-09-30T01:05:34.348 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:05:34.349 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:05:34.349 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:05:34.349 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:05:34.349 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:05:34.349 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-osd.0.asok 2023-09-30T01:05:34.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:05:34.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-osd.0.asok dump_osd_network 2023-09-30T01:05:34.431 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:05:34.432 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 1000, 2023-09-30T01:05:34.432 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [] 2023-09-30T01:05:34.432 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:05:34.440 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:34.441 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:05:34.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:05:34.481 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:34.481 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2023-09-30T01:05:34.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2023-09-30T01:05:34.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:05:34.525 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2023-09-30T01:05:34.525 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2023-09-30T01:05:34.525 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2023-09-30T01:05:34.526 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:05:34.526 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:05:34.526 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:05:34.527 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-mgr.x.asok 2023-09-30T01:05:34.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:05:34.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-mgr.x.asok dump_osd_network 2023-09-30T01:05:34.599 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:05:34.600 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 1000, 2023-09-30T01:05:34.600 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [] 2023-09-30T01:05:34.600 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:05:34.608 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:34.609 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:05:34.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:05:34.648 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:34.648 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2023-09-30T01:05:34.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2023-09-30T01:05:34.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:05:34.680 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2023-09-30T01:05:34.680 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:05:34.680 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:05:34.680 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:05:34.681 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:05:34.681 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:05:34.681 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-osd.0.asok 2023-09-30T01:05:34.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:05:34.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-osd.0.asok dump_osd_network 0 2023-09-30T01:05:34.764 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:05:34.764 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 0, 2023-09-30T01:05:34.764 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [ 2023-09-30T01:05:34.765 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.765 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:34 2023", 2023-09-30T01:05:34.765 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.765 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:34.765 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:34.765 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:34.765 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.765 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.557, 2023-09-30T01:05:34.765 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.572, 2023-09-30T01:05:34.765 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.573 2023-09-30T01:05:34.766 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.766 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.766 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.653, 2023-09-30T01:05:34.766 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.716, 2023-09-30T01:05:34.766 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.724 2023-09-30T01:05:34.766 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.766 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.766 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.653, 2023-09-30T01:05:34.766 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.716, 2023-09-30T01:05:34.766 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.724 2023-09-30T01:05:34.766 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.536 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:34 2023", 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.495, 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.547, 2023-09-30T01:05:34.767 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.547 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.587, 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.666, 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.677 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.587, 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.666, 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.677 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.768 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.415 2023-09-30T01:05:34.769 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.769 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.769 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:34 2023", 2023-09-30T01:05:34.769 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.769 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:34.769 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:34.769 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:34.769 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.769 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.532, 2023-09-30T01:05:34.769 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.545, 2023-09-30T01:05:34.769 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.530 2023-09-30T01:05:34.770 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.770 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.770 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.596, 2023-09-30T01:05:34.770 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.689, 2023-09-30T01:05:34.770 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.793 2023-09-30T01:05:34.770 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.770 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.770 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.596, 2023-09-30T01:05:34.770 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.689, 2023-09-30T01:05:34.770 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.793 2023-09-30T01:05:34.770 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.583 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:34 2023", 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.479, 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.473, 2023-09-30T01:05:34.771 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.473 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.577, 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.590, 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.651 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.577, 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.590, 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.651 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.388 2023-09-30T01:05:34.772 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:05:34.773 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:05:34.773 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:05:34.773 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:34.773 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:05:34.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2023-09-30T01:05:34.814 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:34.814 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2023-09-30T01:05:34.855 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:05:34.856 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:05:34.856 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2023-09-30T01:05:34.857 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2023-09-30T01:05:34.857 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2023-09-30T01:05:34.857 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:05:34.858 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:05:34.858 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:05:34.858 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-mgr.x.asok 2023-09-30T01:05:34.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:05:34.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-mgr.x.asok dump_osd_network 0 2023-09-30T01:05:34.929 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:05:34.930 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 0, 2023-09-30T01:05:34.930 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [ 2023-09-30T01:05:34.930 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.930 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:33 2023", 2023-09-30T01:05:34.930 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.930 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:05:34.931 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:34.931 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:34.931 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.931 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.526, 2023-09-30T01:05:34.931 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.591, 2023-09-30T01:05:34.931 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.592 2023-09-30T01:05:34.931 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.931 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.931 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.410, 2023-09-30T01:05:34.931 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.410, 2023-09-30T01:05:34.932 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.389 2023-09-30T01:05:34.932 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.932 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.932 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.600, 2023-09-30T01:05:34.932 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.728, 2023-09-30T01:05:34.932 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.728 2023-09-30T01:05:34.932 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.932 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.410 2023-09-30T01:05:34.932 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.932 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.932 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:32 2023", 2023-09-30T01:05:34.933 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.933 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:05:34.933 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:34.933 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:34.933 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.933 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.589, 2023-09-30T01:05:34.933 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.586, 2023-09-30T01:05:34.933 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.580 2023-09-30T01:05:34.933 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.933 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.933 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.481, 2023-09-30T01:05:34.934 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.479, 2023-09-30T01:05:34.934 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.442 2023-09-30T01:05:34.934 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.934 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.934 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.678, 2023-09-30T01:05:34.934 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.692, 2023-09-30T01:05:34.934 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.713 2023-09-30T01:05:34.934 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.934 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.640 2023-09-30T01:05:34.934 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.934 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.935 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:29 2023", 2023-09-30T01:05:34.935 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.935 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:34.935 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:34.935 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:34.935 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.935 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.578, 2023-09-30T01:05:34.935 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.573, 2023-09-30T01:05:34.935 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.576 2023-09-30T01:05:34.935 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.550, 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.459, 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.367 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.632, 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.716, 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.724 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.505 2023-09-30T01:05:34.936 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:29 2023", 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.571, 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.552, 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.552 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.937 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.938 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.487, 2023-09-30T01:05:34.938 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.402, 2023-09-30T01:05:34.938 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.264 2023-09-30T01:05:34.938 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.938 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.938 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.656, 2023-09-30T01:05:34.938 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.666, 2023-09-30T01:05:34.938 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.677 2023-09-30T01:05:34.938 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.938 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.462 2023-09-30T01:05:34.938 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.939 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.939 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:32 2023", 2023-09-30T01:05:34.939 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.939 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:05:34.939 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:34.939 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:34.939 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.939 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.501, 2023-09-30T01:05:34.939 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.556, 2023-09-30T01:05:34.939 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.565 2023-09-30T01:05:34.940 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.940 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.940 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.467, 2023-09-30T01:05:34.940 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.467, 2023-09-30T01:05:34.940 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.386 2023-09-30T01:05:34.940 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.940 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.940 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.543, 2023-09-30T01:05:34.940 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.687, 2023-09-30T01:05:34.940 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.687 2023-09-30T01:05:34.941 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.941 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.543 2023-09-30T01:05:34.941 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.941 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.941 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:32 2023", 2023-09-30T01:05:34.941 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.941 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:05:34.941 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:05:34.941 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:34.941 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.941 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.533, 2023-09-30T01:05:34.942 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.550, 2023-09-30T01:05:34.942 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.534 2023-09-30T01:05:34.942 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.942 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.942 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.441, 2023-09-30T01:05:34.942 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.389, 2023-09-30T01:05:34.942 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.389 2023-09-30T01:05:34.942 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.942 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.942 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.589, 2023-09-30T01:05:34.943 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.650, 2023-09-30T01:05:34.943 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.650 2023-09-30T01:05:34.943 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.943 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.581 2023-09-30T01:05:34.943 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.943 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.943 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:33 2023", 2023-09-30T01:05:34.943 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.943 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:05:34.943 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:05:34.943 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:34.944 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.944 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.539, 2023-09-30T01:05:34.944 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.535, 2023-09-30T01:05:34.944 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.549 2023-09-30T01:05:34.944 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.944 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.944 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.440, 2023-09-30T01:05:34.944 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.301, 2023-09-30T01:05:34.944 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.301 2023-09-30T01:05:34.944 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.944 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.945 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.590, 2023-09-30T01:05:34.945 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.624, 2023-09-30T01:05:34.945 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.712 2023-09-30T01:05:34.945 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.945 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.577 2023-09-30T01:05:34.945 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.945 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.945 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:29 2023", 2023-09-30T01:05:34.945 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.945 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:34.945 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:34.946 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:34.946 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.946 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.524, 2023-09-30T01:05:34.946 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.549, 2023-09-30T01:05:34.946 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.530 2023-09-30T01:05:34.946 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.946 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.946 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.490, 2023-09-30T01:05:34.946 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.471, 2023-09-30T01:05:34.946 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.344 2023-09-30T01:05:34.946 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.947 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.947 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.554, 2023-09-30T01:05:34.947 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.689, 2023-09-30T01:05:34.947 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.793 2023-09-30T01:05:34.947 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.947 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.539 2023-09-30T01:05:34.947 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.947 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.947 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:33 2023", 2023-09-30T01:05:34.947 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.947 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.474, 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.516, 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.524 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.352, 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.352, 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.352 2023-09-30T01:05:34.948 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.586, 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.650, 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.756 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.352 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:33 2023", 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:05:34.949 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:05:34.950 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:34.950 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.950 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.521, 2023-09-30T01:05:34.950 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.474, 2023-09-30T01:05:34.950 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.487 2023-09-30T01:05:34.950 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.950 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.951 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.455, 2023-09-30T01:05:34.951 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.333, 2023-09-30T01:05:34.951 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.285 2023-09-30T01:05:34.951 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.951 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.951 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.633, 2023-09-30T01:05:34.951 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.657, 2023-09-30T01:05:34.951 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.657 2023-09-30T01:05:34.951 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.951 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.473 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:32 2023", 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.467, 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.489, 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.488 2023-09-30T01:05:34.952 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.391, 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.391, 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.369 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.522, 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.608, 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.690 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.510 2023-09-30T01:05:34.953 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:05:29 2023", 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.459, 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.470, 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.472 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.954 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.448, 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.312, 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.291 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.472, 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.590, 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.651 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.427 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:05:34.955 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:05:34.956 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:05:34.956 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:34.956 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:05:34.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2023-09-30T01:05:34.978 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2023-09-30T01:05:34.978 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:05:35.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:05:35.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2023-09-30T01:06:25.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2023-09-30T01:06:25.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:06:25.013 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:06:25.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2023-09-30T01:06:25.419 INFO:tasks.workunit.client.0.smithi186.stderr:1 2023-09-30T01:06:25.419 INFO:tasks.workunit.client.0.smithi186.stderr:2' 2023-09-30T01:06:25.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:06:25.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:06:25.419 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:06:25.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836531 2023-09-30T01:06:25.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836531 2023-09-30T01:06:25.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836531' 2023-09-30T01:06:25.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:06:25.553 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-09-30T01:06:25.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949673004 2023-09-30T01:06:25.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949673004 2023-09-30T01:06:25.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836531 1-42949673004' 2023-09-30T01:06:25.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:06:25.695 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-09-30T01:06:25.835 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509476 2023-09-30T01:06:25.835 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509476 2023-09-30T01:06:25.835 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836531 1-42949673004 2-64424509476' 2023-09-30T01:06:25.835 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:06:25.836 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836531 2023-09-30T01:06:25.836 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:06:25.838 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:06:25.838 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836531 2023-09-30T01:06:25.839 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:06:25.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836531 2023-09-30T01:06:25.841 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836531 2023-09-30T01:06:25.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2023-09-30T01:06:25.841 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:06:26.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836528 -lt 21474836531 2023-09-30T01:06:26.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:06:27.285 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:06:27.286 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:06:27.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836531 -lt 21474836531 2023-09-30T01:06:27.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:06:27.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949673004 2023-09-30T01:06:27.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:06:27.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2023-09-30T01:06:27.737 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949673004 2023-09-30T01:06:27.737 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:06:27.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949673004 2023-09-30T01:06:27.739 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.1 seq 42949673004 2023-09-30T01:06:27.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2023-09-30T01:06:27.740 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:06:28.148 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949673004 -lt 42949673004 2023-09-30T01:06:28.148 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:06:28.149 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509476 2023-09-30T01:06:28.149 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:06:28.151 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2023-09-30T01:06:28.153 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509476 2023-09-30T01:06:28.153 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:06:28.154 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.2 seq 64424509476 2023-09-30T01:06:28.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509476 2023-09-30T01:06:28.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2023-09-30T01:06:28.155 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2023-09-30T01:06:28.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509476 -lt 64424509476 2023-09-30T01:06:28.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:06:28.557 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2023-09-30T01:06:28.557 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:06:28.557 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:06:28.557 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:06:28.558 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:28.558 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:28.558 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-osd.0.asok 2023-09-30T01:06:28.558 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:06:28.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-osd.0.asok dump_osd_network 2023-09-30T01:06:28.640 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:06:28.640 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 1000, 2023-09-30T01:06:28.640 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [] 2023-09-30T01:06:28.641 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:06:28.649 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:06:28.650 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:06:28.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:06:28.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:06:28.694 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2023-09-30T01:06:28.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2023-09-30T01:06:28.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:06:28.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2023-09-30T01:06:28.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2023-09-30T01:06:28.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2023-09-30T01:06:28.735 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:06:28.735 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:28.735 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:28.735 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-mgr.x.asok 2023-09-30T01:06:28.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:06:28.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-mgr.x.asok dump_osd_network 2023-09-30T01:06:28.811 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:06:28.811 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 1000, 2023-09-30T01:06:28.811 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [] 2023-09-30T01:06:28.812 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:06:28.820 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:06:28.821 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:06:28.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:06:28.861 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:06:28.861 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2023-09-30T01:06:28.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2023-09-30T01:06:28.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:06:28.903 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2023-09-30T01:06:28.903 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:06:28.903 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:06:28.904 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:06:28.904 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:28.904 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:28.904 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-osd.0.asok 2023-09-30T01:06:28.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:06:28.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-osd.0.asok dump_osd_network 0 2023-09-30T01:06:28.987 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:06:28.987 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 0, 2023-09-30T01:06:28.987 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [ 2023-09-30T01:06:28.987 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:28.988 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:24 2023", 2023-09-30T01:06:28.988 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:28.988 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:06:28.988 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:06:28.988 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:06:28.988 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:28.988 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.572, 2023-09-30T01:06:28.988 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.588, 2023-09-30T01:06:28.988 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.575 2023-09-30T01:06:28.988 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.988 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.669, 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.669, 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.716 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.669, 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.669, 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.716 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.571 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.989 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:28.990 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:24 2023", 2023-09-30T01:06:28.990 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:28.990 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:06:28.990 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:06:28.990 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:06:28.990 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:28.990 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.554, 2023-09-30T01:06:28.990 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.561, 2023-09-30T01:06:28.990 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.558 2023-09-30T01:06:28.990 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.990 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.568, 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.680, 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.680 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.568, 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.680, 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.680 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.513 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.991 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:28.992 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:24 2023", 2023-09-30T01:06:28.992 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:28.992 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:06:28.992 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:06:28.992 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:06:28.992 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:28.992 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.544, 2023-09-30T01:06:28.992 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.544, 2023-09-30T01:06:28.992 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.539 2023-09-30T01:06:28.992 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.992 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.730, 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.730, 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.730 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.730, 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.730, 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.730 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.482 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.993 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:24 2023", 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.521, 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.514, 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.483 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:28.994 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.616, 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.681, 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.681 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.616, 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.681, 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.681 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.361 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:06:28.995 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:06:28.996 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:06:28.997 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:06:29.038 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2023-09-30T01:06:29.039 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:06:29.039 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2023-09-30T01:06:29.082 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:06:29.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:06:29.084 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2023-09-30T01:06:29.084 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2023-09-30T01:06:29.084 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2023-09-30T01:06:29.085 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:06:29.085 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:29.085 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:29.085 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-mgr.x.asok 2023-09-30T01:06:29.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:06:29.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-mgr.x.asok dump_osd_network 0 2023-09-30T01:06:29.161 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:06:29.161 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 0, 2023-09-30T01:06:29.161 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [ 2023-09-30T01:06:29.161 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.161 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:23 2023", 2023-09-30T01:06:29.161 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.161 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:06:29.161 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:06:29.162 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:06:29.162 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.162 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.572, 2023-09-30T01:06:29.162 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.587, 2023-09-30T01:06:29.162 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.594 2023-09-30T01:06:29.162 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.162 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.162 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.502, 2023-09-30T01:06:29.162 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.502, 2023-09-30T01:06:29.162 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.386 2023-09-30T01:06:29.162 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.163 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.163 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.724, 2023-09-30T01:06:29.163 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.724, 2023-09-30T01:06:29.163 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.728 2023-09-30T01:06:29.163 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.163 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.439 2023-09-30T01:06:29.163 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.163 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.163 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:24 2023", 2023-09-30T01:06:29.163 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.163 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:06:29.164 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:06:29.164 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:06:29.164 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.164 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.572, 2023-09-30T01:06:29.164 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.588, 2023-09-30T01:06:29.164 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.575 2023-09-30T01:06:29.164 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.164 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.164 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.483, 2023-09-30T01:06:29.164 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.483, 2023-09-30T01:06:29.164 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.459 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.669, 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.669, 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.716 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.461 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:27 2023", 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.165 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:06:29.166 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:06:29.166 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:06:29.166 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.166 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.536, 2023-09-30T01:06:29.166 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.572, 2023-09-30T01:06:29.166 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.575 2023-09-30T01:06:29.166 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.166 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.166 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.429, 2023-09-30T01:06:29.166 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.429, 2023-09-30T01:06:29.166 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.370 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.638, 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.645, 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.698 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.548 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:24 2023", 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.167 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:06:29.168 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:06:29.168 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:06:29.168 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.168 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.554, 2023-09-30T01:06:29.168 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.561, 2023-09-30T01:06:29.168 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.558 2023-09-30T01:06:29.168 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.168 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.168 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.526, 2023-09-30T01:06:29.168 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.425, 2023-09-30T01:06:29.168 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.415 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.568, 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.680, 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.680 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.516 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:23 2023", 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.169 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.556, 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.552, 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.529 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.479, 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.479, 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.352 2023-09-30T01:06:29.170 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.614, 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.641, 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.673 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.575 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:24 2023", 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:06:29.171 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.544, 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.544, 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.539 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.451, 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.451, 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.391 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.172 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.730, 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.730, 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.730 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.731 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:27 2023", 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:06:29.173 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "front", 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.524, 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.542, 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.543 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.458, 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.431, 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.431 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.174 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.594, 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.643, 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.801 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.488 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:27 2023", 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:06:29.175 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.494, 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.529, 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.539 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.331, 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.331, 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.331 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.582, 2023-09-30T01:06:29.176 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.651, 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.751 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.515 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:23 2023", 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 2, 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.177 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.512, 2023-09-30T01:06:29.178 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.523, 2023-09-30T01:06:29.178 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.531 2023-09-30T01:06:29.178 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.178 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.178 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.473, 2023-09-30T01:06:29.178 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.442, 2023-09-30T01:06:29.178 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.301 2023-09-30T01:06:29.178 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.178 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.178 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.560, 2023-09-30T01:06:29.178 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.578, 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.625 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.269 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:24 2023", 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 0, 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 1, 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.179 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.521, 2023-09-30T01:06:29.180 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.514, 2023-09-30T01:06:29.180 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.483 2023-09-30T01:06:29.180 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.181 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.181 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.443, 2023-09-30T01:06:29.181 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.403, 2023-09-30T01:06:29.181 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.388 2023-09-30T01:06:29.181 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.181 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.181 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.616, 2023-09-30T01:06:29.181 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.681, 2023-09-30T01:06:29.181 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.681 2023-09-30T01:06:29.181 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.181 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.360 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:23 2023", 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 1, 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.520, 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.481, 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.474 2023-09-30T01:06:29.182 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.391, 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.391, 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.333 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.631, 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.631, 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.657 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.363 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.183 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:06:29.184 INFO:tasks.workunit.client.0.smithi186.stdout: "last update": "Sat Sep 30 01:06:27 2023", 2023-09-30T01:06:29.184 INFO:tasks.workunit.client.0.smithi186.stdout: "stale": false, 2023-09-30T01:06:29.184 INFO:tasks.workunit.client.0.smithi186.stdout: "from osd": 2, 2023-09-30T01:06:29.184 INFO:tasks.workunit.client.0.smithi186.stdout: "to osd": 0, 2023-09-30T01:06:29.184 INFO:tasks.workunit.client.0.smithi186.stdout: "interface": "back", 2023-09-30T01:06:29.184 INFO:tasks.workunit.client.0.smithi186.stdout: "average": { 2023-09-30T01:06:29.184 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.477, 2023-09-30T01:06:29.184 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.475, 2023-09-30T01:06:29.185 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.473 2023-09-30T01:06:29.185 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.185 INFO:tasks.workunit.client.0.smithi186.stdout: "min": { 2023-09-30T01:06:29.185 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.343, 2023-09-30T01:06:29.185 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.343, 2023-09-30T01:06:29.185 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.343 2023-09-30T01:06:29.185 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.186 INFO:tasks.workunit.client.0.smithi186.stdout: "max": { 2023-09-30T01:06:29.186 INFO:tasks.workunit.client.0.smithi186.stdout: "1min": 0.634, 2023-09-30T01:06:29.186 INFO:tasks.workunit.client.0.smithi186.stdout: "5min": 0.634, 2023-09-30T01:06:29.186 INFO:tasks.workunit.client.0.smithi186.stdout: "15min": 0.634 2023-09-30T01:06:29.186 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:06:29.186 INFO:tasks.workunit.client.0.smithi186.stdout: "last": 0.465 2023-09-30T01:06:29.186 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:06:29.186 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:06:29.186 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:06:29.187 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:06:29.187 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2023-09-30T01:06:29.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2023-09-30T01:06:29.212 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:06:29.212 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2023-09-30T01:06:29.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2023-09-30T01:06:29.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:06:29.255 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2023-09-30T01:06:29.255 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2023-09-30T01:06:29.255 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2023-09-30T01:06:29.256 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:06:29.256 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:29.256 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:29.256 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-mgr.x.asok 2023-09-30T01:06:29.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:06:29.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-mgr.x.asok dump_osd_network 99 2023-09-30T01:06:29.330 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:06:29.330 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 99, 2023-09-30T01:06:29.330 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [] 2023-09-30T01:06:29.331 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:06:29.338 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:06:29.338 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2023-09-30T01:06:29.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2023-09-30T01:06:29.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2023-09-30T01:06:29.379 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2023-09-30T01:06:29.379 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:06:29.379 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:06:29.380 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:06:29.380 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:29.380 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:29.380 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-osd.0.asok 2023-09-30T01:06:29.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2023-09-30T01:06:29.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.40580/ceph-osd.0.asok dump_osd_network 98 2023-09-30T01:06:29.461 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:06:29.461 INFO:tasks.workunit.client.0.smithi186.stdout: "threshold": 98, 2023-09-30T01:06:29.461 INFO:tasks.workunit.client.0.smithi186.stdout: "entries": [] 2023-09-30T01:06:29.462 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:06:29.469 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2023-09-30T01:06:29.470 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2023-09-30T01:06:29.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2023-09-30T01:06:29.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2023-09-30T01:06:29.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2023-09-30T01:06:29.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2023-09-30T01:06:29.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:06:29.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2023-09-30T01:06:29.514 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:06:29.514 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:06:29.515 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:06:29.515 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:06:29.515 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:06:29.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:06:29.631 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:06:29.633 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:06:29.633 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:06:29.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:06:29.636 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:06:29.636 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:06:29.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:06:29.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:06:29.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:06:29.639 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:06:29.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:06:29.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:06:29.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2023-09-30T01:06:29.666 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:06:29.666 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:29.666 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:29.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.40580 2023-09-30T01:06:29.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:06:29.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:06:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2023-09-30T01:06:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2023-09-30T01:06:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2023-09-30T01:06:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2023-09-30T01:06:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2023-09-30T01:06:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:06:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2023-09-30T01:06:29.670 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:06:29.670 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:06:29.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:06:29.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:06:29.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:06:29.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:06:29.674 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:06:29.675 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:06:29.676 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:06:29.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:06:29.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:06:29.679 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:06:29.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:06:29.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:06:29.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:06:29.681 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:06:29.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:06:29.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:06:29.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2023-09-30T01:06:29.686 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:06:29.687 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:29.687 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:29.687 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.40580 2023-09-30T01:06:29.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:06:29.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:06:29.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2023-09-30T01:06:29.690 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:06:29.691 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:29.691 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:29.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.40580 2023-09-30T01:06:29.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:06:29.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:06:29.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:06:29.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2023-09-30T01:06:29.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2023-09-30T01:06:29.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2023-09-30T01:06:29.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2023-09-30T01:06:29.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2023-09-30T01:06:29.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2023-09-30T01:06:29.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2023-09-30T01:06:29.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2023-09-30T01:06:29.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:06:29.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:06:29.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:06:29.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2023-09-30T01:06:29.697 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2023-09-30T01:06:30.096 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:06:30.096 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:06:30.096 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:06:30.097 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:06:30.097 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:30.097 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:30.098 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:06:30.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:06:30.361 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:06:30.363 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:06:30.363 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:06:30.363 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:06:30.363 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:06:30.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:06:30.365 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:06:30.365 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:06:30.365 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:06:30.365 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:06:30.366 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:30.366 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:30.366 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-mon.a.asok 2023-09-30T01:06:30.366 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:06:30.366 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.40580/ceph-mon.a.asok config get fsid 2023-09-30T01:06:30.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:06:30.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:06:30.460 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:06:30.460 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:06:30.460 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:06:30.461 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:06:30.461 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:06:30.461 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:06:30.461 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:06:30.461 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:30.461 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:30.461 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.40580/ceph-mon.a.asok 2023-09-30T01:06:30.461 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:06:30.462 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.40580/ceph-mon.a.asok config get mon_host 2023-09-30T01:06:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2023-09-30T01:06:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2023-09-30T01:06:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:06:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:06:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:06:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2023-09-30T01:06:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:06:31.441 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:06:31.441 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:06:31.441 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:06:31.441 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:06:31.442 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:31.442 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:31.442 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:06:31.442 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:06:31.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:06:31.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2023-09-30T01:06:31.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2023-09-30T01:06:31.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:06:31.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:06:31.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:06:31.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2023-09-30T01:06:31.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2023-09-30T01:06:31.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:06:31.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:06:31.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:06:31.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2023-09-30T01:06:31.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2023-09-30T01:06:31.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:06:31.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2023-09-30T01:06:31.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2023-09-30T01:06:31.476 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:06:31.477 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:06:31.477 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:06:31.478 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:06:31.478 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:06:31.478 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:06:31.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:06:31.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:06:31.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2023-09-30T01:06:31.480 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:06:31.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f5996079-56a9-448a-938f-42404a1ba03c 2023-09-30T01:06:31.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f5996079-56a9-448a-938f-42404a1ba03c' 2023-09-30T01:06:31.481 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 f5996079-56a9-448a-938f-42404a1ba03c 2023-09-30T01:06:31.481 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:06:31.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXdBdl4/6hHRAAh20PorsRYAV9l1T9BAEu0A== 2023-09-30T01:06:31.499 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXdBdl4/6hHRAAh20PorsRYAV9l1T9BAEu0A=="}' 2023-09-30T01:06:31.499 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f5996079-56a9-448a-938f-42404a1ba03c -i td/network-ping/0/new.json 2023-09-30T01:06:31.877 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:06:31.892 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2023-09-30T01:06:31.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXdBdl4/6hHRAAh20PorsRYAV9l1T9BAEu0A== --osd-uuid f5996079-56a9-448a-938f-42404a1ba03c 2023-09-30T01:06:31.926 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:31.921+0000 7f4ac2b263c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:06:31.929 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:31.925+0000 7f4ac2b263c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:06:31.931 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:31.925+0000 7f4ac2b263c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:06:31.932 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:31.925+0000 7f4ac2b263c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2023-09-30T01:06:31.932 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:31.925+0000 7f4ac2b263c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2023-09-30T01:06:31.932 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:31.925+0000 7f4ac2b263c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2023-09-30T01:06:31.985 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:31.981+0000 7f4ac2b263c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2023-09-30T01:06:38.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2023-09-30T01:06:38.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:06:38.362 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:06:38.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:06:38.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:06:38.955 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:06:38.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:06:38.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:06:38.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:06:38.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:06:38.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:06:38.988 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:38.984+0000 7ffb56fc53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:06:39.020 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:39.012+0000 7ffb56fc53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:06:39.021 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:39.016+0000 7ffb56fc53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:06:39.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:06:39.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:06:39.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:06:39.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:06:39.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:06:39.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:39.405 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:06:39.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:06:39.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:39.406 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:39.610 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:39.604+0000 7ffb56fc53c0 -1 Falling back to public interface 2023-09-30T01:06:39.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:40.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:40.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:40.817 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:06:40.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:06:40.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:40.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:41.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:41.881 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:06:41.876+0000 7ffb56fc53c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:06:42.233 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:42.233 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:42.233 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:06:42.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:06:42.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:42.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:42.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:43.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:43.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:43.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:06:43.772 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:06:43.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:43.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:44.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:45.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:45.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:45.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:06:45.238 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:06:45.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:45.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:45.777 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:46.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:46.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:46.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:06:46.779 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:06:46.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:46.780 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:47.259 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:48.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:48.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:48.262 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:06:48.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:06:48.263 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:48.263 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:48.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:49.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:49.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:49.711 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:06:49.711 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:06:49.711 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:49.711 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:50.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:51.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:51.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:51.177 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:06:51.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:06:51.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:51.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:51.700 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:52.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:52.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:52.702 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:06:52.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:06:52.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:52.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:53.194 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:54.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:54.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:54.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:06:54.197 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:06:54.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:54.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:54.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:55.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:55.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:55.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:06:55.710 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:06:55.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:55.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:56.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:57.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:57.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:57.228 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:06:57.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:06:57.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:57.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:58.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:06:59.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:06:59.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:06:59.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:06:59.136 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:06:59.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:06:59.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:06:59.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:00.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:00.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:00.667 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:07:00.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:07:00.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:00.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:07:01.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:02.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:02.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:02.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:07:02.525 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:07:02.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:02.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:07:03.421 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:04.423 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:04.423 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:04.423 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:07:04.423 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:07:04.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:04.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:07:04.879 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/48088,v1:127.0.0.1:6803/48088] [v2:127.0.0.1:6804/48088,v1:127.0.0.1:6805/48088] exists,up f5996079-56a9-448a-938f-42404a1ba03c 2023-09-30T01:07:04.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:07:04.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:07:04.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:07:04.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2023-09-30T01:07:04.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2023-09-30T01:07:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:07:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-09-30T01:07:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:07:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2023-09-30T01:07:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2023-09-30T01:07:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:07:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:07:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:07:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2023-09-30T01:07:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2023-09-30T01:07:04.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:07:04.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2023-09-30T01:07:04.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2023-09-30T01:07:04.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:07:04.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:07:04.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:07:04.883 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:07:04.883 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:07:04.884 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:07:04.884 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:07:04.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:07:04.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:07:04.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:07:04.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:07:04.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2023-09-30T01:07:04.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2023-09-30T01:07:04.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:07:04.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:07:04.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:07:04.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:07:04.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:07:04.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:07:04.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2023-09-30T01:07:04.885 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:07:04.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62e6cdaf-9595-4ae1-beb8-3b943642575f 2023-09-30T01:07:04.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 62e6cdaf-9595-4ae1-beb8-3b943642575f' 2023-09-30T01:07:04.887 INFO:tasks.workunit.client.0.smithi186.stdout:add osd1 62e6cdaf-9595-4ae1-beb8-3b943642575f 2023-09-30T01:07:04.887 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:07:04.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4dBdlx97bNRAAHWRZ8XogyYYW97RR0YJAxw== 2023-09-30T01:07:04.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4dBdlx97bNRAAHWRZ8XogyYYW97RR0YJAxw=="}' 2023-09-30T01:07:04.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62e6cdaf-9595-4ae1-beb8-3b943642575f -i td/network-ping/1/new.json 2023-09-30T01:07:05.457 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:07:05.470 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2023-09-30T01:07:05.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4dBdlx97bNRAAHWRZ8XogyYYW97RR0YJAxw== --osd-uuid 62e6cdaf-9595-4ae1-beb8-3b943642575f 2023-09-30T01:07:05.508 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:05.500+0000 7f5653f993c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:05.509 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:05.504+0000 7f5653f993c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:05.511 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:05.504+0000 7f5653f993c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:05.512 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:05.504+0000 7f5653f993c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2023-09-30T01:07:05.512 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:05.504+0000 7f5653f993c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2023-09-30T01:07:05.512 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:05.504+0000 7f5653f993c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2023-09-30T01:07:05.616 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:05.608+0000 7f5653f993c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2023-09-30T01:07:11.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2023-09-30T01:07:11.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:07:11.540 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd1 key to auth repository 2023-09-30T01:07:11.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-09-30T01:07:11.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:07:12.099 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-09-30T01:07:12.100 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:07:12.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:07:12.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:07:12.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:07:12.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:07:12.124 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:12.116+0000 7f979e4d63c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:12.126 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:12.120+0000 7f979e4d63c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:12.128 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:12.120+0000 7f979e4d63c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:12.411 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:12.404+0000 7f979e4d63c0 -1 Falling back to public interface 2023-09-30T01:07:12.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-09-30T01:07:12.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:07:12.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:07:12.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:07:12.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:07:12.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:12.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:07:12.553 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:07:12.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:12.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:12.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:13.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:13.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:13.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:07:13.957 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:07:13.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:13.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:14.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:15.072 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:15.064+0000 7f979e4d63c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:07:15.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:15.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:15.368 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:07:15.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:07:15.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:15.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:15.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:16.931 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:16.931 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:16.932 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:07:16.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:07:16.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:16.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:17.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:18.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:18.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:18.448 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:07:18.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:07:18.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:18.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:18.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:19.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:19.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:19.970 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:07:19.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:07:19.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:19.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:20.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:21.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:21.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:21.418 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:07:21.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:07:21.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:21.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:21.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:22.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:22.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:22.893 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:07:22.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:07:22.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:22.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:23.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:24.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:24.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:24.311 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:07:24.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:07:24.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:24.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:24.824 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:25.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:25.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:25.828 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:07:25.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:07:25.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:25.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:26.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:27.230 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:27.230 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:27.230 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:07:27.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:07:27.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:27.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:28.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:29.478 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:29.478 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:29.479 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:07:29.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:07:29.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:29.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:30.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:31.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:31.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:31.203 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:07:31.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:07:31.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:31.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:31.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:32.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:32.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:32.972 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:07:32.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:07:32.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:32.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:34.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:35.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:35.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:35.048 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:07:35.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:07:35.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:35.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:35.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:36.568 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:36.568 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:36.569 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:07:36.569 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:07:36.569 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:36.569 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:37.301 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:38.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:38.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:38.303 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:07:38.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:07:38.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:38.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:07:38.691 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/49836,v1:127.0.0.1:6811/49836] [v2:127.0.0.1:6812/49836,v1:127.0.0.1:6813/49836] exists,up 62e6cdaf-9595-4ae1-beb8-3b943642575f 2023-09-30T01:07:38.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:07:38.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:07:38.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:07:38.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2023-09-30T01:07:38.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2023-09-30T01:07:38.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:07:38.694 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-09-30T01:07:38.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:07:38.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2023-09-30T01:07:38.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2023-09-30T01:07:38.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:07:38.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:07:38.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:07:38.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2023-09-30T01:07:38.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2023-09-30T01:07:38.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:07:38.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2023-09-30T01:07:38.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2023-09-30T01:07:38.696 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:07:38.696 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:07:38.696 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:07:38.698 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:07:38.698 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:07:38.698 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:07:38.698 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:07:38.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' 2023-09-30T01:07:38.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:07:38.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:07:38.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:07:38.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2023-09-30T01:07:38.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2023-09-30T01:07:38.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:07:38.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:07:38.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:07:38.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:07:38.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:07:38.700 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:07:38.700 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2023-09-30T01:07:38.700 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:07:38.700 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b666b23f-c8ba-43e0-affb-923408d0e8ff 2023-09-30T01:07:38.700 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b666b23f-c8ba-43e0-affb-923408d0e8ff' 2023-09-30T01:07:38.701 INFO:tasks.workunit.client.0.smithi186.stdout:add osd2 b666b23f-c8ba-43e0-affb-923408d0e8ff 2023-09-30T01:07:38.701 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:07:38.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDadBdl3RD+KhAAdJF/TYZOt4hELt/fsBDcqw== 2023-09-30T01:07:38.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDadBdl3RD+KhAAdJF/TYZOt4hELt/fsBDcqw=="}' 2023-09-30T01:07:38.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b666b23f-c8ba-43e0-affb-923408d0e8ff -i td/network-ping/2/new.json 2023-09-30T01:07:39.260 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:07:39.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2023-09-30T01:07:39.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDadBdl3RD+KhAAdJF/TYZOt4hELt/fsBDcqw== --osd-uuid b666b23f-c8ba-43e0-affb-923408d0e8ff 2023-09-30T01:07:39.313 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:39.311+0000 7fb2144733c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:39.341 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:39.339+0000 7fb2144733c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:39.343 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:39.339+0000 7fb2144733c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:39.344 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:39.339+0000 7fb2144733c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2023-09-30T01:07:39.344 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:39.339+0000 7fb2144733c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2023-09-30T01:07:39.344 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:39.339+0000 7fb2144733c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2023-09-30T01:07:39.458 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:39.455+0000 7fb2144733c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2023-09-30T01:07:45.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2023-09-30T01:07:45.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:07:45.605 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd2 key to auth repository 2023-09-30T01:07:45.605 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-09-30T01:07:45.605 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:07:46.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-09-30T01:07:46.145 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.2 2023-09-30T01:07:46.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=15254a43-07b8-40da-9ccd-bf524680b6c0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.40580/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:07:46.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:07:46.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:07:46.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:07:46.171 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:46.167+0000 7fe0bf4823c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:46.173 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:46.171+0000 7fe0bf4823c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:46.176 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:46.171+0000 7fe0bf4823c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:07:46.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-09-30T01:07:46.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:07:46.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-09-30T01:07:46.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:07:46.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:07:46.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:46.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:07:46.604 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:07:46.605 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:46.605 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:07:46.723 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:46.719+0000 7fe0bf4823c0 -1 Falling back to public interface 2023-09-30T01:07:47.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:48.043 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:48.043 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:48.044 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:07:48.044 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:07:48.044 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:48.044 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:07:48.455 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:49.253 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:07:49.251+0000 7fe0bf4823c0 -1 osd.2 0 log_to_monitors true 2023-09-30T01:07:49.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:49.458 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:49.458 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:07:49.458 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:07:49.459 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:49.459 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:07:49.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:50.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:50.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:50.955 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:07:50.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:07:50.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:50.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:07:51.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:52.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:52.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:52.400 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:07:52.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:07:52.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:52.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:07:52.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:53.847 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:53.847 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:53.847 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:07:53.848 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:07:53.848 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:53.848 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:07:54.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:55.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:55.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:55.324 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:07:55.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:07:55.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:55.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:07:55.777 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:56.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:56.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:56.780 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:07:56.780 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:07:56.780 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:56.780 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:07:57.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:58.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:58.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:58.349 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:07:58.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:07:58.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:07:58.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:07:58.892 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:07:59.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:07:59.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:07:59.894 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:07:59.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:07:59.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:07:59.895 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:08:00.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:08:01.286 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:08:01.286 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:08:01.286 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:08:01.286 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:08:01.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:08:01.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:08:01.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:08:02.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:08:02.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:08:02.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:08:02.988 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:08:02.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:08:02.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:08:03.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:08:04.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:08:04.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:08:04.947 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:08:04.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:08:04.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:08:04.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:08:05.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:08:06.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:08:06.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:08:06.530 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:08:06.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:08:06.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:08:06.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:08:07.468 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:08:08.470 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:08:08.470 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:08:08.471 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:08:08.471 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:08:08.471 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:08:08.471 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:08:09.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:08:10.030 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:08:10.030 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:08:10.030 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:08:10.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:08:10.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:08:10.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:08:11.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:08:12.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:08:12.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:08:12.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:08:12.016 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:08:12.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:08:12.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:08:12.464 INFO:tasks.workunit.client.0.smithi186.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/51585,v1:127.0.0.1:6819/51585] [v2:127.0.0.1:6820/51585,v1:127.0.0.1:6821/51585] exists,up b666b23f-c8ba-43e0-affb-923408d0e8ff 2023-09-30T01:08:12.465 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:08:12.465 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:08:12.465 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:08:12.465 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2023-09-30T01:08:17.466 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2023-09-30T01:08:18.034 INFO:tasks.workunit.client.0.smithi186.stderr:bucket 'dc1' already exists 2023-09-30T01:08:18.044 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2023-09-30T01:08:18.614 INFO:tasks.workunit.client.0.smithi186.stderr:bucket 'dc2' already exists 2023-09-30T01:08:18.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2023-09-30T01:08:19.341 INFO:tasks.workunit.client.0.smithi186.stderr:bucket 'dc3' already exists 2023-09-30T01:08:19.351 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2023-09-30T01:08:19.995 INFO:tasks.workunit.client.0.smithi186.stderr:bucket 'rack1' already exists 2023-09-30T01:08:20.003 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2023-09-30T01:08:20.621 INFO:tasks.workunit.client.0.smithi186.stderr:bucket 'rack2' already exists 2023-09-30T01:08:20.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2023-09-30T01:08:21.211 INFO:tasks.workunit.client.0.smithi186.stderr:bucket 'rack3' already exists 2023-09-30T01:08:21.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2023-09-30T01:08:21.840 INFO:tasks.workunit.client.0.smithi186.stderr:bucket 'host1' already exists 2023-09-30T01:08:21.848 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2023-09-30T01:08:22.572 INFO:tasks.workunit.client.0.smithi186.stderr:bucket 'host2' already exists 2023-09-30T01:08:22.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2023-09-30T01:08:23.190 INFO:tasks.workunit.client.0.smithi186.stderr:bucket 'host3' already exists 2023-09-30T01:08:23.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2023-09-30T01:08:23.941 INFO:tasks.workunit.client.0.smithi186.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2023-09-30T01:08:23.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2023-09-30T01:08:24.583 INFO:tasks.workunit.client.0.smithi186.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2023-09-30T01:08:24.597 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2023-09-30T01:08:25.211 INFO:tasks.workunit.client.0.smithi186.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2023-09-30T01:08:25.223 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2023-09-30T01:08:25.834 INFO:tasks.workunit.client.0.smithi186.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2023-09-30T01:08:25.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2023-09-30T01:08:26.459 INFO:tasks.workunit.client.0.smithi186.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2023-09-30T01:08:26.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2023-09-30T01:08:27.163 INFO:tasks.workunit.client.0.smithi186.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2023-09-30T01:08:27.171 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2023-09-30T01:08:27.830 INFO:tasks.workunit.client.0.smithi186.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2023-09-30T01:08:27.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2023-09-30T01:08:28.383 INFO:tasks.workunit.client.0.smithi186.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2023-09-30T01:08:28.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2023-09-30T01:08:29.024 INFO:tasks.workunit.client.0.smithi186.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2023-09-30T01:08:29.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2023-09-30T01:08:29.624 INFO:tasks.workunit.client.0.smithi186.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2023-09-30T01:08:29.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2023-09-30T01:08:30.218 INFO:tasks.workunit.client.0.smithi186.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2023-09-30T01:08:30.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2023-09-30T01:08:30.949 INFO:tasks.workunit.client.0.smithi186.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2023-09-30T01:08:30.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2023-09-30T01:08:31.587 INFO:tasks.workunit.client.0.smithi186.stderr:rule myrule already exists 2023-09-30T01:08:31.597 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2023-09-30T01:08:31.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2023-09-30T01:08:32.208 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'foo' already exists 2023-09-30T01:08:32.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:08:33.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2023-09-30T01:08:33.263 INFO:tasks.workunit.client.0.smithi186.stdout:hints = 1 2023-09-30T01:08:33.263 INFO:tasks.workunit.client.0.smithi186.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2023-09-30T01:08:33.263 INFO:tasks.workunit.client.0.smithi186.stdout:Object prefix: benchmark_data_smithi186_53049 2023-09-30T01:08:33.263 INFO:tasks.workunit.client.0.smithi186.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2023-09-30T01:08:33.263 INFO:tasks.workunit.client.0.smithi186.stdout: 0 0 0 0 0 0 - 0 2023-09-30T01:08:34.262 INFO:tasks.workunit.client.0.smithi186.stdout: 1 16 47 31 0.121082 0.121094 0.286161 0.448181 2023-09-30T01:08:35.262 INFO:tasks.workunit.client.0.smithi186.stdout: 2 16 95 79 0.154277 0.1875 0.239389 0.389214 2023-09-30T01:08:36.263 INFO:tasks.workunit.client.0.smithi186.stdout: 3 16 159 143 0.186175 0.25 0.26209 0.333187 2023-09-30T01:08:37.263 INFO:tasks.workunit.client.0.smithi186.stdout: 4 16 208 192 0.187476 0.191406 0.265008 0.316634 2023-09-30T01:08:38.263 INFO:tasks.workunit.client.0.smithi186.stdout: 5 16 271 255 0.199193 0.246094 0.321542 0.305056 2023-09-30T01:08:39.263 INFO:tasks.workunit.client.0.smithi186.stdout: 6 16 335 319 0.207655 0.25 0.253944 0.29648 2023-09-30T01:08:40.264 INFO:tasks.workunit.client.0.smithi186.stdout: 7 16 384 368 0.20533 0.191406 0.292571 0.297116 2023-09-30T01:08:41.263 INFO:tasks.workunit.client.0.smithi186.stdout: 8 16 447 431 0.21042 0.246094 0.278345 0.292757 2023-09-30T01:08:42.263 INFO:tasks.workunit.client.0.smithi186.stdout: 9 16 511 495 0.214813 0.25 0.310618 0.290049 2023-09-30T01:08:43.264 INFO:tasks.workunit.client.0.smithi186.stdout: 10 16 560 544 0.21247 0.191406 0.267249 0.28839 2023-09-30T01:08:44.264 INFO:tasks.workunit.client.0.smithi186.stdout:Total time run: 10.2138 2023-09-30T01:08:44.264 INFO:tasks.workunit.client.0.smithi186.stdout:Total writes made: 560 2023-09-30T01:08:44.264 INFO:tasks.workunit.client.0.smithi186.stdout:Write size: 4096 2023-09-30T01:08:44.264 INFO:tasks.workunit.client.0.smithi186.stdout:Object size: 4096 2023-09-30T01:08:44.264 INFO:tasks.workunit.client.0.smithi186.stdout:Bandwidth (MB/sec): 0.214172 2023-09-30T01:08:44.264 INFO:tasks.workunit.client.0.smithi186.stdout:Stddev Bandwidth: 0.0431931 2023-09-30T01:08:44.265 INFO:tasks.workunit.client.0.smithi186.stdout:Max bandwidth (MB/sec): 0.25 2023-09-30T01:08:44.265 INFO:tasks.workunit.client.0.smithi186.stdout:Min bandwidth (MB/sec): 0.121094 2023-09-30T01:08:44.265 INFO:tasks.workunit.client.0.smithi186.stdout:Average IOPS: 54 2023-09-30T01:08:44.265 INFO:tasks.workunit.client.0.smithi186.stdout:Stddev IOPS: 11.0574 2023-09-30T01:08:44.265 INFO:tasks.workunit.client.0.smithi186.stdout:Max IOPS: 64 2023-09-30T01:08:44.265 INFO:tasks.workunit.client.0.smithi186.stdout:Min IOPS: 31 2023-09-30T01:08:44.265 INFO:tasks.workunit.client.0.smithi186.stdout:Average Latency(s): 0.289094 2023-09-30T01:08:44.265 INFO:tasks.workunit.client.0.smithi186.stdout:Stddev Latency(s): 0.0744474 2023-09-30T01:08:44.265 INFO:tasks.workunit.client.0.smithi186.stdout:Max latency(s): 0.684695 2023-09-30T01:08:44.265 INFO:tasks.workunit.client.0.smithi186.stdout:Min latency(s): 0.224875 2023-09-30T01:08:44.351 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2023-09-30T01:08:54.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2023-09-30T01:08:54.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:08:54.354 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:08:54.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2023-09-30T01:08:54.774 INFO:tasks.workunit.client.0.smithi186.stderr:1 2023-09-30T01:08:54.774 INFO:tasks.workunit.client.0.smithi186.stderr:2' 2023-09-30T01:08:54.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:08:54.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:08:54.775 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:08:54.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836513 2023-09-30T01:08:54.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836513 2023-09-30T01:08:54.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836513' 2023-09-30T01:08:54.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:08:54.960 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-09-30T01:08:55.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672984 2023-09-30T01:08:55.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672984 2023-09-30T01:08:55.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836513 1-42949672984' 2023-09-30T01:08:55.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:08:55.101 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-09-30T01:08:55.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509460 2023-09-30T01:08:55.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509460 2023-09-30T01:08:55.251 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836513 1-42949672984 2-64424509460' 2023-09-30T01:08:55.251 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:08:55.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836513 2023-09-30T01:08:55.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:08:55.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:08:55.252 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836513 2023-09-30T01:08:55.252 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:08:55.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836513 2023-09-30T01:08:55.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2023-09-30T01:08:55.254 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836513 2023-09-30T01:08:55.254 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:08:55.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836511 -lt 21474836513 2023-09-30T01:08:55.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:08:56.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:08:56.665 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:08:57.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836513 -lt 21474836513 2023-09-30T01:08:57.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:08:57.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672984 2023-09-30T01:08:57.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:08:57.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2023-09-30T01:08:57.063 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672984 2023-09-30T01:08:57.063 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:08:57.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672984 2023-09-30T01:08:57.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2023-09-30T01:08:57.065 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.1 seq 42949672984 2023-09-30T01:08:57.065 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:08:57.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672984 -lt 42949672984 2023-09-30T01:08:57.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:08:57.474 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509460 2023-09-30T01:08:57.475 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:08:57.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2023-09-30T01:08:57.478 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:08:57.478 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509460 2023-09-30T01:08:57.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509460 2023-09-30T01:08:57.479 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.2 seq 64424509460 2023-09-30T01:08:57.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2023-09-30T01:08:57.480 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2023-09-30T01:08:57.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509460 -lt 64424509460 2023-09-30T01:08:57.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2023-09-30T01:08:57.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2023-09-30T01:08:58.394 INFO:tasks.workunit.client.0.smithi186.stdout:HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'; Slow OSD heartbeats on back (longest 0.577ms); Slow OSD heartbeats on front (longest 0.605ms); 1 pool(s) do not have an application enabled 2023-09-30T01:08:58.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2023-09-30T01:08:58.408 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2023-09-30T01:08:58.408 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2023-09-30T01:08:58.884 INFO:tasks.workunit.client.0.smithi186.stdout:HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'; Slow OSD heartbeats on back (longest 0.565ms); Slow OSD heartbeats on front (longest 0.605ms); 1 pool(s) do not have an application enabled 2023-09-30T01:08:58.884 INFO:tasks.workunit.client.0.smithi186.stdout:[ERR] MGR_MODULE_ERROR: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:08:58.884 INFO:tasks.workunit.client.0.smithi186.stdout: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:08:58.885 INFO:tasks.workunit.client.0.smithi186.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.565ms) 2023-09-30T01:08:58.885 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.565 msec 2023-09-30T01:08:58.885 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.561 msec possibly improving 2023-09-30T01:08:58.885 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.547 msec 2023-09-30T01:08:58.885 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.532 msec 2023-09-30T01:08:58.885 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.513 msec possibly improving 2023-09-30T01:08:58.885 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.506 msec possibly improving 2023-09-30T01:08:58.885 INFO:tasks.workunit.client.0.smithi186.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.605ms) 2023-09-30T01:08:58.885 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.605 msec 2023-09-30T01:08:58.885 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.587 msec 2023-09-30T01:08:58.885 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.571 msec 2023-09-30T01:08:58.886 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.562 msec 2023-09-30T01:08:58.886 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.558 msec 2023-09-30T01:08:58.886 INFO:tasks.workunit.client.0.smithi186.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.510 msec possibly improving 2023-09-30T01:08:58.886 INFO:tasks.workunit.client.0.smithi186.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2023-09-30T01:08:58.886 INFO:tasks.workunit.client.0.smithi186.stdout: application not enabled on pool 'foo' 2023-09-30T01:08:58.886 INFO:tasks.workunit.client.0.smithi186.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2023-09-30T01:08:58.895 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2023-09-30T01:08:58.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2023-09-30T01:08:58.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2023-09-30T01:08:58.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2023-09-30T01:08:58.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2023-09-30T01:08:58.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2023-09-30T01:08:58.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:08:58.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2023-09-30T01:08:58.903 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:08:58.904 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:08:58.904 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:08:58.904 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:08:58.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:08:59.020 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:08:59.021 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:08:59.023 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:08:59.023 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:08:59.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:08:59.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:08:59.028 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:08:59.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:08:59.029 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:08:59.029 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:08:59.030 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:08:59.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:08:59.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:08:59.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2023-09-30T01:08:59.056 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:08:59.056 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:08:59.056 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:08:59.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.40580 2023-09-30T01:08:59.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:08:59.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:08:59.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: main: code=0 2023-09-30T01:08:59.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: main: teardown td/network-ping 0 2023-09-30T01:08:59.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2023-09-30T01:08:59.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-09-30T01:08:59.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2023-09-30T01:08:59.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:08:59.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:08:59.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:08:59.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:08:59.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:08:59.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:08:59.064 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:08:59.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:08:59.066 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:08:59.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:08:59.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:08:59.068 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:08:59.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:08:59.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:08:59.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:08:59.071 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:08:59.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:08:59.075 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-09-30T01:08:59.075 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2023-09-30T01:08:59.077 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:08:59.077 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:08:59.077 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.40580 2023-09-30T01:08:59.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.40580 2023-09-30T01:08:59.079 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:08:59.079 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:08:59.079 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: main: return 0 2023-09-30T01:08:59.080 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-30T01:08:59.080 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-30T01:08:59.094 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2023-09-30T01:08:59.094 DEBUG:teuthology.orchestra.run.smithi186:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=93dce5e17918f8aca65993a7a2dc40233a47b481 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2023-09-30T01:08:59.155 INFO:tasks.workunit.client.0.smithi186.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-09-30T01:08:59.171 INFO:tasks.workunit.client.0.smithi186.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-09-30T01:08:59.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-09-30T01:08:59.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-09-30T01:08:59.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: export PYTHONWARNINGS=ignore 2023-09-30T01:08:59.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: PYTHONWARNINGS=ignore 2023-09-30T01:08:59.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: export CEPH_CONF=/dev/null 2023-09-30T01:08:59.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: CEPH_CONF=/dev/null 2023-09-30T01:08:59.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: unset CEPH_ARGS 2023-09-30T01:08:59.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: main: local code 2023-09-30T01:08:59.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: main: run td/ok-to-stop 2023-09-30T01:08:59.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2023-09-30T01:08:59.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2023-09-30T01:08:59.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2023-09-30T01:08:59.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2023-09-30T01:08:59.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2023-09-30T01:08:59.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2023-09-30T01:08:59.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2023-09-30T01:08:59.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2023-09-30T01:08:59.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2023-09-30T01:08:59.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2023-09-30T01:08:59.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2023-09-30T01:08:59.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2023-09-30T01:08:59.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2023-09-30T01:08:59.175 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2023-09-30T01:08:59.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none ' 2023-09-30T01:08:59.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none ' 2023-09-30T01:08:59.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none ' 2023-09-30T01:08:59.177 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2023-09-30T01:08:59.177 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-09-30T01:08:59.179 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2023-09-30T01:08:59.179 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_0_osd 2023-09-30T01:08:59.179 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_1_mon_checks 2023-09-30T01:08:59.179 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_2_mons_checks 2023-09-30T01:08:59.179 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_3_mons_checks 2023-09-30T01:08:59.179 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_4_mons_checks 2023-09-30T01:08:59.179 INFO:tasks.workunit.client.0.smithi186.stderr:TEST_5_mons_checks' 2023-09-30T01:08:59.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2023-09-30T01:08:59.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2023-09-30T01:08:59.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2023-09-30T01:08:59.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2023-09-30T01:08:59.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:08:59.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:08:59.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:08:59.181 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:08:59.181 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:08:59.181 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:08:59.181 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:08:59.181 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:08:59.183 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:08:59.183 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:08:59.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:08:59.185 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:08:59.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:08:59.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:08:59.188 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:08:59.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:08:59.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:08:59.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:08:59.190 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:08:59.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:08:59.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:08:59.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:08:59.195 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:08:59.195 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:08:59.196 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:08:59.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:08:59.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:08:59.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:08:59.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2023-09-30T01:08:59.200 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:08:59.200 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:08:59.200 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:08:59.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53393 2023-09-30T01:08:59.203 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:08:59.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-09-30T01:08:59.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-09-30T01:08:59.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:08:59.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2023-09-30T01:08:59.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2023-09-30T01:08:59.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2023-09-30T01:08:59.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2023-09-30T01:08:59.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2023-09-30T01:08:59.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:08:59.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:08:59.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:08:59.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:08:59.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2023-09-30T01:08:59.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2023-09-30T01:08:59.604 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:08:59.604 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:08:59.604 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:08:59.605 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:08:59.605 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:08:59.605 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:08:59.605 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:08:59.605 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2023-09-30T01:08:59.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:08:59.876 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:08:59.876 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:08:59.876 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:08:59.877 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:08:59.877 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:08:59.878 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:08:59.878 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:08:59.878 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:08:59.878 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:08:59.879 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:08:59.879 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:08:59.879 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:08:59.880 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:08:59.880 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get fsid 2023-09-30T01:08:59.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:08:59.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:08:59.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:08:59.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:08:59.973 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:08:59.973 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:08:59.973 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:08:59.973 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:08:59.974 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:08:59.974 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:08:59.974 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:08:59.974 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:08:59.975 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:08:59.975 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get mon_host 2023-09-30T01:09:00.057 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2023-09-30T01:09:00.057 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2023-09-30T01:09:00.057 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:09:00.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:09:00.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:09:00.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2023-09-30T01:09:00.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:09:00.956 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:09:00.956 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:09:00.956 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:09:00.956 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:09:00.956 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:00.956 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:00.956 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:09:00.957 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:09:00.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:09:00.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2023-09-30T01:09:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2023-09-30T01:09:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:09:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:09:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:09:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2023-09-30T01:09:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2023-09-30T01:09:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:09:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:09:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:09:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2023-09-30T01:09:00.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2023-09-30T01:09:00.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:09:00.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:09:00.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2023-09-30T01:09:00.992 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:09:00.992 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:09:00.992 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:09:00.992 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:09:00.992 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:00.992 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:00.992 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:09:00.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:09:00.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:09:00.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:09:00.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:09:00.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2023-09-30T01:09:00.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2023-09-30T01:09:00.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:09:00.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:09:00.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:09:00.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:09:00.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:09:00.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:09:00.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2023-09-30T01:09:00.995 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:09:00.996 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=058e712c-481f-4df7-ba2e-612208776cdd 2023-09-30T01:09:00.996 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 058e712c-481f-4df7-ba2e-612208776cdd' 2023-09-30T01:09:00.997 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 058e712c-481f-4df7-ba2e-612208776cdd 2023-09-30T01:09:00.998 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:09:01.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtdRdlWl6RABAAtKBFqNiT4iXmcKBrME0eDA== 2023-09-30T01:09:01.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtdRdlWl6RABAAtKBFqNiT4iXmcKBrME0eDA=="}' 2023-09-30T01:09:01.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 058e712c-481f-4df7-ba2e-612208776cdd -i td/ok-to-stop/0/new.json 2023-09-30T01:09:01.374 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:09:01.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2023-09-30T01:09:01.390 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtdRdlWl6RABAAtKBFqNiT4iXmcKBrME0eDA== --osd-uuid 058e712c-481f-4df7-ba2e-612208776cdd 2023-09-30T01:09:01.416 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:01.409+0000 7f51d39353c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:09:01.420 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:01.413+0000 7f51d39353c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:09:01.421 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:01.417+0000 7f51d39353c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:09:01.422 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:01.417+0000 7f51d39353c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2023-09-30T01:09:01.422 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:01.417+0000 7f51d39353c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2023-09-30T01:09:01.422 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:01.417+0000 7f51d39353c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2023-09-30T01:09:01.497 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:01.489+0000 7f51d39353c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2023-09-30T01:09:07.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2023-09-30T01:09:07.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:09:07.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:09:07.746 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:09:07.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:09:08.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:09:08.251 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:09:08.251 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:09:08.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:09:08.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:09:08.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:09:08.280 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:08.273+0000 7f20348643c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:09:08.287 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:08.281+0000 7f20348643c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:09:08.288 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:08.281+0000 7f20348643c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:09:08.602 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:09:08.602 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:09:08.602 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:09:08.602 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:09:08.602 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:09:08.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:08.605 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:09:08.605 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:09:08.606 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:08.606 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:08.843 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:08.837+0000 7f20348643c0 -1 Falling back to public interface 2023-09-30T01:09:09.027 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:10.029 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:10.029 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:10.029 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:09:10.030 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:09:10.030 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:10.030 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:10.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:11.362 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:11.357+0000 7f20348643c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:09:11.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:11.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:11.417 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:09:11.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:09:11.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:11.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:11.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:12.951 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:12.951 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:12.951 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:09:12.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:09:12.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:12.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:13.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:14.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:14.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:14.403 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:09:14.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:09:14.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:14.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:14.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:15.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:15.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:15.791 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:09:15.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:09:15.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:15.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:16.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:17.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:17.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:17.198 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:09:17.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:09:17.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:17.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:17.637 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:18.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:18.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:18.642 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:09:18.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:09:18.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:18.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:19.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:20.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:20.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:20.199 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:09:20.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:09:20.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:20.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:20.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:21.600 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:21.600 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:21.600 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:09:21.601 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:09:21.601 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:21.601 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:22.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:23.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:23.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:23.048 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:09:23.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:09:23.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:23.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:23.904 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:24.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:24.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:24.906 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:09:24.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:09:24.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:24.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:25.515 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:26.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:26.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:26.517 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:09:26.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:09:26.518 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:26.518 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:26.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:27.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:27.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:27.961 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:09:27.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:09:27.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:27.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:28.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:29.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:29.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:29.539 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:09:29.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:09:29.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:29.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:30.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:31.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:31.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:31.731 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:09:31.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:09:31.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:31.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:32.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:09:33.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:09:33.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:09:33.669 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:09:33.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:09:33.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:09:33.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:09:34.089 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/54514,v1:127.0.0.1:6803/54514] [v2:127.0.0.1:6804/54514,v1:127.0.0.1:6805/54514] exists,up 058e712c-481f-4df7-ba2e-612208776cdd 2023-09-30T01:09:34.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:09:34.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:09:34.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:09:34.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2023-09-30T01:09:34.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2023-09-30T01:09:34.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2023-09-30T01:09:34.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2023-09-30T01:09:34.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2023-09-30T01:09:34.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2023-09-30T01:09:34.091 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2023-09-30T01:09:34.092 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:09:34.092 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:09:34.092 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:09:34.093 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:34.093 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:34.093 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:09:34.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2023-09-30T01:09:34.135 INFO:tasks.workunit.client.0.smithi186.stdout:starting mds.a at 2023-09-30T01:09:34.140 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2023-09-30T01:09:34.895 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'meta' already exists 2023-09-30T01:09:34.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2023-09-30T01:09:35.749 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'data' already exists 2023-09-30T01:09:35.761 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2023-09-30T01:09:36.633 INFO:tasks.workunit.client.0.smithi186.stderr:filesystem 'myfs' already exists 2023-09-30T01:09:36.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2023-09-30T01:09:41.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2023-09-30T01:09:42.009 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4151 and/or standby gids 2023-09-30T01:09:42.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2023-09-30T01:09:42.351 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4151 and/or standby gids 2023-09-30T01:09:42.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2023-09-30T01:09:42.881 INFO:tasks.workunit.client.0.smithi186.stderr:should be safe to stop [dne] 2023-09-30T01:09:42.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2023-09-30T01:09:42.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2023-09-30T01:09:42.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2023-09-30T01:09:42.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2023-09-30T01:09:42.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2023-09-30T01:09:42.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2023-09-30T01:09:42.891 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2023-09-30T01:09:42.892 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:09:42.892 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:09:42.892 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:09:42.892 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:42.892 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:42.892 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:09:42.892 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2023-09-30T01:09:42.915 INFO:tasks.workunit.client.0.smithi186.stdout:starting mds.b at 2023-09-30T01:09:42.921 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2023-09-30T01:09:47.922 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2023-09-30T01:09:48.446 INFO:tasks.workunit.client.0.smithi186.stderr:should be safe to stop [a] 2023-09-30T01:09:48.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2023-09-30T01:09:48.988 INFO:tasks.workunit.client.0.smithi186.stderr:should be safe to stop [b] 2023-09-30T01:09:49.003 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2023-09-30T01:09:49.382 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4151 and/or standby gids 4165 2023-09-30T01:09:49.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2023-09-30T01:09:49.942 INFO:tasks.workunit.client.0.smithi186.stderr:should be safe to stop [a,dne1,dne2] 2023-09-30T01:09:49.950 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2023-09-30T01:09:50.491 INFO:tasks.workunit.client.0.smithi186.stderr:should be safe to stop [b,dne] 2023-09-30T01:09:50.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2023-09-30T01:09:50.924 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4151 and/or standby gids 4165 2023-09-30T01:09:50.927 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2023-09-30T01:09:51.410 INFO:tasks.workunit.client.0.smithi186.stderr:should be safe to stop [dne1,dne2] 2023-09-30T01:09:51.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2023-09-30T01:09:51.418 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:09:51.418 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:09:51.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:09:51.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:09:51.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:09:51.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:09:51.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2023-09-30T01:09:51.532 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:09:51.532 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:09:51.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:09:51.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:09:51.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:09:51.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:09:51.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2023-09-30T01:09:51.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:09:51.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:09:51.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:09:51.648 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:09:51.649 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:09:51.649 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:09:51.649 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:09:51.649 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:09:51.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:09:51.661 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:09:51.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:09:51.662 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:09:51.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:09:51.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:09:51.664 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:09:51.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:09:51.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:09:51.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:09:51.666 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:09:51.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:09:51.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:09:51.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:09:51.676 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:09:51.676 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:51.676 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:51.677 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:09:51.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:09:51.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:09:51.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2023-09-30T01:09:51.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2023-09-30T01:09:51.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2023-09-30T01:09:51.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2023-09-30T01:09:51.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:09:51.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:09:51.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:09:51.679 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:09:51.679 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:09:51.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:09:51.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:09:51.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:09:51.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:09:51.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:09:51.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:09:51.685 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:09:51.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:09:51.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:09:51.686 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:09:51.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:09:51.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:09:51.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:09:51.689 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:09:51.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:09:51.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:09:51.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:09:51.694 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:09:51.694 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:51.695 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:51.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:09:51.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:09:51.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:09:51.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2023-09-30T01:09:51.699 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:09:51.699 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:51.699 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:51.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53393 2023-09-30T01:09:51.702 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:09:51.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:09:51.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:09:51.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2023-09-30T01:09:51.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2023-09-30T01:09:51.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2023-09-30T01:09:51.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2023-09-30T01:09:51.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2023-09-30T01:09:51.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:09:51.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:09:51.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:09:51.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:09:51.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2023-09-30T01:09:51.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2023-09-30T01:09:52.110 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:09:52.110 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:09:52.110 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:09:52.112 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:09:52.112 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:52.112 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:52.112 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:09:52.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2023-09-30T01:09:52.398 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:09:52.398 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:09:52.399 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:09:52.399 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:09:52.399 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:09:52.399 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:09:52.400 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:09:52.400 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:09:52.400 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:09:52.400 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:09:52.401 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:52.401 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:52.401 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:09:52.401 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:09:52.402 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get fsid 2023-09-30T01:09:52.486 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:09:52.486 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:09:52.487 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:09:52.487 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:09:52.487 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:09:52.487 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:09:52.488 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:09:52.488 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:09:52.488 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:09:52.488 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:52.488 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:52.488 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:09:52.488 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:09:52.489 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get mon_host 2023-09-30T01:09:52.571 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2023-09-30T01:09:52.572 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2023-09-30T01:09:52.572 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:09:52.572 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:09:52.572 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:09:52.572 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2023-09-30T01:09:52.572 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:09:53.482 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:09:53.482 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:09:53.482 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:09:53.483 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:09:53.483 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:53.483 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:53.483 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:09:53.484 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:09:53.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:09:53.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2023-09-30T01:09:53.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2023-09-30T01:09:53.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:09:53.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:09:53.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:09:53.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2023-09-30T01:09:53.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2023-09-30T01:09:53.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:09:53.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:09:53.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:09:53.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2023-09-30T01:09:53.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2023-09-30T01:09:53.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:09:53.521 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:09:53.521 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2023-09-30T01:09:53.521 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:09:53.521 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:09:53.521 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:09:53.521 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:09:53.521 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:09:53.521 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:09:53.523 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:09:53.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:09:53.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:09:53.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:09:53.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:09:53.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2023-09-30T01:09:53.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2023-09-30T01:09:53.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:09:53.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:09:53.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:09:53.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:09:53.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:09:53.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:09:53.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2023-09-30T01:09:53.524 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:09:53.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=20c8854f-ef2e-4fa5-ab15-27eae2c23513 2023-09-30T01:09:53.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 20c8854f-ef2e-4fa5-ab15-27eae2c23513' 2023-09-30T01:09:53.525 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 20c8854f-ef2e-4fa5-ab15-27eae2c23513 2023-09-30T01:09:53.525 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:09:53.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhdRdlHRE5IBAArWg7xC3cd1aHUTa1Dc3HuQ== 2023-09-30T01:09:53.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhdRdlHRE5IBAArWg7xC3cd1aHUTa1Dc3HuQ=="}' 2023-09-30T01:09:53.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 20c8854f-ef2e-4fa5-ab15-27eae2c23513 -i td/ok-to-stop/0/new.json 2023-09-30T01:09:53.892 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:09:53.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2023-09-30T01:09:53.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhdRdlHRE5IBAArWg7xC3cd1aHUTa1Dc3HuQ== --osd-uuid 20c8854f-ef2e-4fa5-ab15-27eae2c23513 2023-09-30T01:09:53.933 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:53.928+0000 7f20751743c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:09:53.940 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:53.928+0000 7f20751743c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:09:53.940 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:53.932+0000 7f20751743c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:09:53.940 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:53.932+0000 7f20751743c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2023-09-30T01:09:53.940 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:53.932+0000 7f20751743c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2023-09-30T01:09:53.940 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:53.932+0000 7f20751743c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2023-09-30T01:09:53.983 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:09:53.976+0000 7f20751743c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2023-09-30T01:10:00.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2023-09-30T01:10:00.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:10:00.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:10:00.309 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:10:00.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:10:00.841 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:10:00.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:10:00.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:10:00.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:10:00.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:10:00.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:10:00.863 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:00.856+0000 7fa269a953c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:10:00.866 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:00.860+0000 7fa269a953c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:10:00.867 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:00.860+0000 7fa269a953c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:10:01.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:10:01.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:10:01.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:10:01.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:10:01.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:10:01.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:01.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:10:01.243 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:10:01.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:01.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:01.427 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:01.420+0000 7fa269a953c0 -1 Falling back to public interface 2023-09-30T01:10:01.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:02.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:02.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:02.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:10:02.733 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:10:02.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:02.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:03.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:03.946 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:03.940+0000 7fa269a953c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:10:04.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:04.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:04.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:10:04.118 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:10:04.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:04.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:04.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:05.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:05.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:05.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:10:05.643 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:10:05.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:05.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:06.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:07.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:07.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:07.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:10:07.072 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:10:07.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:07.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:07.470 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:08.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:08.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:08.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:10:08.473 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:10:08.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:08.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:08.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:09.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:09.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:09.982 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:10:09.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:10:09.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:09.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:10.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:11.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:11.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:11.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:10:11.413 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:10:11.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:11.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:11.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:12.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:12.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:12.933 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:10:12.933 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:10:12.933 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:12.933 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:13.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:14.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:14.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:14.335 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:10:14.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:10:14.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:14.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:15.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:16.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:16.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:16.217 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:10:16.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:10:16.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:16.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:17.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:18.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:18.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:18.017 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:10:18.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:10:18.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:18.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:18.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:19.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:19.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:19.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:10:19.855 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:10:19.855 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:19.855 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:20.916 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:21.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:21.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:21.918 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:10:21.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:10:21.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:21.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:23.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:24.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:24.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:24.133 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:10:24.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:10:24.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:24.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:25.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:26.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:26.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:26.033 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:10:26.034 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:10:26.034 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:26.034 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:10:26.485 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56972,v1:127.0.0.1:6803/56972] [v2:127.0.0.1:6804/56972,v1:127.0.0.1:6805/56972] exists,up 20c8854f-ef2e-4fa5-ab15-27eae2c23513 2023-09-30T01:10:26.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:10:26.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:10:26.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:10:26.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2023-09-30T01:10:26.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2023-09-30T01:10:26.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:10:26.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-09-30T01:10:26.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:10:26.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2023-09-30T01:10:26.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2023-09-30T01:10:26.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:10:26.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:10:26.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:10:26.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2023-09-30T01:10:26.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2023-09-30T01:10:26.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:10:26.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:10:26.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2023-09-30T01:10:26.490 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:10:26.490 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:10:26.490 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:10:26.490 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:10:26.490 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:10:26.490 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:10:26.492 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:10:26.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:10:26.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:10:26.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:10:26.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:10:26.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2023-09-30T01:10:26.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2023-09-30T01:10:26.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:10:26.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:10:26.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:10:26.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:10:26.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:10:26.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:10:26.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2023-09-30T01:10:26.529 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:10:26.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9871812c-0289-4f8e-b783-3fa9c810cd0e 2023-09-30T01:10:26.531 INFO:tasks.workunit.client.0.smithi186.stdout:add osd1 9871812c-0289-4f8e-b783-3fa9c810cd0e 2023-09-30T01:10:26.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9871812c-0289-4f8e-b783-3fa9c810cd0e' 2023-09-30T01:10:26.531 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:10:26.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCdRdlJIyoIBAAatgbRNfP/tcUxMJ9iULtCA== 2023-09-30T01:10:26.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCdRdlJIyoIBAAatgbRNfP/tcUxMJ9iULtCA=="}' 2023-09-30T01:10:26.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9871812c-0289-4f8e-b783-3fa9c810cd0e -i td/ok-to-stop/1/new.json 2023-09-30T01:10:27.070 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:10:27.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2023-09-30T01:10:27.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCdRdlJIyoIBAAatgbRNfP/tcUxMJ9iULtCA== --osd-uuid 9871812c-0289-4f8e-b783-3fa9c810cd0e 2023-09-30T01:10:27.108 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:27.103+0000 7f55c3bf23c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:10:27.125 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:27.123+0000 7f55c3bf23c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:10:27.127 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:27.123+0000 7f55c3bf23c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:10:27.128 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:27.123+0000 7f55c3bf23c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2023-09-30T01:10:27.128 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:27.123+0000 7f55c3bf23c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2023-09-30T01:10:27.128 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:27.123+0000 7f55c3bf23c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2023-09-30T01:10:27.217 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:27.211+0000 7f55c3bf23c0 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2023-09-30T01:10:33.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2023-09-30T01:10:33.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:10:33.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-09-30T01:10:33.553 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd1 key to auth repository 2023-09-30T01:10:33.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:10:34.112 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:10:34.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-09-30T01:10:34.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:10:34.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:10:34.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:10:34.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:10:34.140 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:34.135+0000 7fb7dfdbf3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:10:34.146 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:34.143+0000 7fb7dfdbf3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:10:34.148 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:34.143+0000 7fb7dfdbf3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:10:34.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-09-30T01:10:34.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:10:34.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:10:34.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:10:34.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:10:34.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:34.494 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:10:34.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:10:34.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:34.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:34.696 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:34.691+0000 7fb7dfdbf3c0 -1 Falling back to public interface 2023-09-30T01:10:34.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:35.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:35.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:35.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:10:35.947 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:10:35.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:35.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:36.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:37.229 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:10:37.223+0000 7fb7dfdbf3c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:10:37.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:37.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:37.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:10:37.353 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:10:37.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:37.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:37.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:38.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:38.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:38.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:10:38.886 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:10:38.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:38.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:39.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:40.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:40.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:40.353 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:10:40.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:10:40.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:40.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:40.761 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:41.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:41.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:41.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:10:41.763 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:10:41.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:41.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:42.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:43.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:43.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:43.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:10:43.306 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:10:43.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:43.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:43.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:44.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:44.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:44.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:10:44.711 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:10:44.711 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:44.711 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:45.109 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:46.111 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:46.111 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:46.111 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:10:46.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:10:46.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:46.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:46.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:47.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:47.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:47.508 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:10:47.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:10:47.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:47.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:47.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:48.981 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:48.981 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:48.981 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:10:48.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:10:48.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:48.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:50.055 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:51.057 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:51.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:51.058 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:10:51.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:10:51.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:51.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:51.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:52.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:52.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:52.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:10:52.682 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:10:52.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:52.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:53.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:54.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:54.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:54.318 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:10:54.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:10:54.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:54.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:55.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:56.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:56.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:56.198 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:10:56.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:10:56.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:56.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:57.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:58.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:58.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:58.004 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:10:58.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:10:58.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:58.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:10:58.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:10:59.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:10:59.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:10:59.800 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:10:59.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:10:59.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:10:59.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:11:00.206 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/58695,v1:127.0.0.1:6811/58695] [v2:127.0.0.1:6812/58695,v1:127.0.0.1:6813/58695] exists,up 9871812c-0289-4f8e-b783-3fa9c810cd0e 2023-09-30T01:11:00.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:11:00.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:11:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:11:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2023-09-30T01:11:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2023-09-30T01:11:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:11:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-09-30T01:11:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:11:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2023-09-30T01:11:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2023-09-30T01:11:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:11:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:11:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:11:00.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2023-09-30T01:11:00.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2023-09-30T01:11:00.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:11:00.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:11:00.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2023-09-30T01:11:00.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:11:00.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:11:00.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:11:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:11:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:11:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:11:00.212 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:11:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:11:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:11:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:11:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:11:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2023-09-30T01:11:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2023-09-30T01:11:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:11:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:11:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:11:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:11:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:11:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:11:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2023-09-30T01:11:00.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:11:00.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4c6c1e8f-9199-4618-be76-38fe1c87aad6 2023-09-30T01:11:00.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4c6c1e8f-9199-4618-be76-38fe1c87aad6' 2023-09-30T01:11:00.215 INFO:tasks.workunit.client.0.smithi186.stdout:add osd2 4c6c1e8f-9199-4618-be76-38fe1c87aad6 2023-09-30T01:11:00.216 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:11:00.233 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCkdRdlXrvKDRAABDdyN1wqLxQ7bjLgDTXGZg== 2023-09-30T01:11:00.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCkdRdlXrvKDRAABDdyN1wqLxQ7bjLgDTXGZg=="}' 2023-09-30T01:11:00.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4c6c1e8f-9199-4618-be76-38fe1c87aad6 -i td/ok-to-stop/2/new.json 2023-09-30T01:11:00.752 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:11:00.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2023-09-30T01:11:00.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCkdRdlXrvKDRAABDdyN1wqLxQ7bjLgDTXGZg== --osd-uuid 4c6c1e8f-9199-4618-be76-38fe1c87aad6 2023-09-30T01:11:00.799 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:00.795+0000 7f56df10b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:00.802 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:00.799+0000 7f56df10b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:00.804 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:00.799+0000 7f56df10b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:00.805 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:00.803+0000 7f56df10b3c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2023-09-30T01:11:00.806 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:00.803+0000 7f56df10b3c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2023-09-30T01:11:00.806 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:00.803+0000 7f56df10b3c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2023-09-30T01:11:00.938 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:00.935+0000 7f56df10b3c0 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2023-09-30T01:11:06.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2023-09-30T01:11:06.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:11:06.920 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-09-30T01:11:06.921 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd2 key to auth repository 2023-09-30T01:11:06.921 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:11:07.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-09-30T01:11:07.448 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.2 2023-09-30T01:11:07.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:11:07.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:11:07.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:11:07.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:11:07.477 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:07.470+0000 7fa836ff73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:07.479 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:07.474+0000 7fa836ff73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:07.481 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:07.474+0000 7fa836ff73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:07.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-09-30T01:11:07.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:11:07.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-09-30T01:11:07.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:11:07.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:11:07.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:07.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:11:07.891 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:11:07.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:07.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:08.032 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:08.026+0000 7fa836ff73c0 -1 Falling back to public interface 2023-09-30T01:11:08.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:09.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:09.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:09.269 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:11:09.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:11:09.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:09.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:09.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:10.610 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:10.606+0000 7fa836ff73c0 -1 osd.2 0 log_to_monitors true 2023-09-30T01:11:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:11:10.672 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:11:10.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:10.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:11.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:12.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:12.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:12.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:11:12.199 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:11:12.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:12.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:12.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:13.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:13.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:13.616 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:11:13.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:11:13.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:13.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:14.038 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:15.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:15.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:15.040 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:11:15.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:11:15.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:15.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:15.501 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:16.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:16.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:16.504 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:11:16.504 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:11:16.504 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:16.505 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:16.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:17.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:17.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:17.970 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:11:17.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:11:17.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:17.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:18.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:19.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:19.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:19.513 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:11:19.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:11:19.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:19.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:19.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:20.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:20.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:20.962 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:11:20.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:11:20.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:20.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:21.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:22.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:22.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:22.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:11:22.498 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:11:22.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:22.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:23.434 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:24.436 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:24.436 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:24.437 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:11:24.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:11:24.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:24.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:24.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:25.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:25.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:25.962 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:11:25.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:11:25.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:25.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:26.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:27.831 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:27.831 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:27.831 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:11:27.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:11:27.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:27.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:28.848 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:29.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:29.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:29.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:11:29.851 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:11:29.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:29.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:30.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:31.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:31.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:31.733 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:11:31.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:11:31.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:31.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:32.474 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:33.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:33.477 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:33.477 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:11:33.477 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:11:33.478 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:33.478 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:11:33.997 INFO:tasks.workunit.client.0.smithi186.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/60445,v1:127.0.0.1:6819/60445] [v2:127.0.0.1:6820/60445,v1:127.0.0.1:6821/60445] exists,up 4c6c1e8f-9199-4618-be76-38fe1c87aad6 2023-09-30T01:11:33.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:11:33.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:11:33.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:11:33.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2023-09-30T01:11:33.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2023-09-30T01:11:33.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:11:33.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-09-30T01:11:33.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:11:33.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2023-09-30T01:11:33.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2023-09-30T01:11:33.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:11:33.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:11:33.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:11:33.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2023-09-30T01:11:34.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2023-09-30T01:11:34.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:11:34.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:11:34.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2023-09-30T01:11:34.000 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:11:34.000 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:11:34.000 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:11:34.001 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:11:34.001 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:11:34.001 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:11:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:11:34.003 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:11:34.003 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2023-09-30T01:11:34.003 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:11:34.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=46c94cee-682b-4cce-b1c0-82aa02e14fe1 2023-09-30T01:11:34.004 INFO:tasks.workunit.client.0.smithi186.stdout:add osd3 46c94cee-682b-4cce-b1c0-82aa02e14fe1 2023-09-30T01:11:34.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 46c94cee-682b-4cce-b1c0-82aa02e14fe1' 2023-09-30T01:11:34.005 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:11:34.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDGdRdlpDf9ABAADHUNmHjWNKESKX6YZi8Ehw== 2023-09-30T01:11:34.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDGdRdlpDf9ABAADHUNmHjWNKESKX6YZi8Ehw=="}' 2023-09-30T01:11:34.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 46c94cee-682b-4cce-b1c0-82aa02e14fe1 -i td/ok-to-stop/3/new.json 2023-09-30T01:11:34.568 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:11:34.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2023-09-30T01:11:34.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDGdRdlpDf9ABAADHUNmHjWNKESKX6YZi8Ehw== --osd-uuid 46c94cee-682b-4cce-b1c0-82aa02e14fe1 2023-09-30T01:11:34.616 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:34.610+0000 7f23829de3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:34.619 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:34.614+0000 7f23829de3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:34.621 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:34.614+0000 7f23829de3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:34.622 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:34.618+0000 7f23829de3c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2023-09-30T01:11:34.622 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:34.618+0000 7f23829de3c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2023-09-30T01:11:34.622 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:34.618+0000 7f23829de3c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2023-09-30T01:11:34.700 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:34.694+0000 7f23829de3c0 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2023-09-30T01:11:40.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2023-09-30T01:11:40.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:11:40.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2023-09-30T01:11:40.736 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd3 key to auth repository 2023-09-30T01:11:40.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:11:41.267 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.3 2023-09-30T01:11:41.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2023-09-30T01:11:41.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:11:41.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:11:41.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:11:41.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:11:41.297 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:41.290+0000 7f167a3db3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:41.299 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:41.294+0000 7f167a3db3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:41.301 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:41.294+0000 7f167a3db3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:11:41.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2023-09-30T01:11:41.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:11:41.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-09-30T01:11:41.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:11:41.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:11:41.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:41.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:11:41.749 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:11:41.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:41.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:41.860 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:41.854+0000 7f167a3db3c0 -1 Falling back to public interface 2023-09-30T01:11:42.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:43.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:43.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:43.155 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:11:43.156 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:11:43.156 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:43.156 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:43.636 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:44.459 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:11:44.454+0000 7f167a3db3c0 -1 osd.3 0 log_to_monitors true 2023-09-30T01:11:44.637 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:44.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:44.638 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:11:44.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:11:44.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:44.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:45.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:46.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:46.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:46.228 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:11:46.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:11:46.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:46.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:46.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:47.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:47.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:47.654 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:11:47.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:11:47.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:47.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:48.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:49.123 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:49.123 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:49.124 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:11:49.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:11:49.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:49.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:49.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:50.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:50.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:50.628 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:11:50.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:11:50.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:50.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:51.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:52.027 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:52.027 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:52.027 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:11:52.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:11:52.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:52.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:52.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:53.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:53.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:53.538 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:11:53.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:11:53.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:53.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:53.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:54.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:54.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:54.990 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:11:54.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:11:54.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:54.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:55.521 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:56.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:56.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:56.523 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:11:56.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:11:56.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:56.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:57.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:11:58.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:11:58.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:11:58.311 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:11:58.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:11:58.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:11:58.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:11:59.182 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:12:00.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:12:00.185 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:12:00.185 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:12:00.185 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:12:00.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:12:00.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:12:00.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:12:01.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:12:01.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:12:01.905 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:12:01.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:12:01.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:12:01.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:12:02.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:12:03.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:12:03.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:12:03.725 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:12:03.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:12:03.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:12:03.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:12:04.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:12:05.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:12:05.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:12:05.533 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:12:05.534 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:12:05.534 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:12:05.534 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:12:06.079 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:12:07.081 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:12:07.081 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:12:07.081 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:12:07.082 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:12:07.082 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:12:07.082 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-09-30T01:12:07.569 INFO:tasks.workunit.client.0.smithi186.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/62194,v1:127.0.0.1:6827/62194] [v2:127.0.0.1:6828/62194,v1:127.0.0.1:6829/62194] exists,up 46c94cee-682b-4cce-b1c0-82aa02e14fe1 2023-09-30T01:12:07.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:12:07.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:12:07.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:12:07.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2023-09-30T01:12:08.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2023-09-30T01:12:09.008 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'ec' already exists 2023-09-30T01:12:09.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2023-09-30T01:12:09.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:12:09.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:12:09.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:12:09.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:12:09.017 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:12:09.018 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:12:09.018 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:12:09.018 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:12:09.018 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:12:09.018 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:12:09.219 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:12:09.219 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:12:09.219 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:12:09.219 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:12:09.219 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:12:09.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2023-09-30T01:12:09.632 INFO:tasks.workunit.client.0.smithi186.stderr:1 2023-09-30T01:12:09.632 INFO:tasks.workunit.client.0.smithi186.stderr:2 2023-09-30T01:12:09.632 INFO:tasks.workunit.client.0.smithi186.stderr:3' 2023-09-30T01:12:09.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:12:09.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:09.632 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:12:10.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836504 2023-09-30T01:12:10.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836504 2023-09-30T01:12:10.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836504' 2023-09-30T01:12:10.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:10.015 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-09-30T01:12:10.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672990 2023-09-30T01:12:10.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672990 2023-09-30T01:12:10.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836504 1-42949672990' 2023-09-30T01:12:10.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:10.235 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-09-30T01:12:10.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509460 2023-09-30T01:12:10.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509460 2023-09-30T01:12:10.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836504 1-42949672990 2-64424509460' 2023-09-30T01:12:10.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:10.566 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-09-30T01:12:10.836 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=85899345930 2023-09-30T01:12:10.836 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 85899345930 2023-09-30T01:12:10.836 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836504 1-42949672990 2-64424509460 3-85899345930' 2023-09-30T01:12:10.836 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:10.837 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836504 2023-09-30T01:12:10.837 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:10.838 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:12:10.839 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836504 2023-09-30T01:12:10.839 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:10.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836504 2023-09-30T01:12:10.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2023-09-30T01:12:10.842 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836504 2023-09-30T01:12:10.842 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:12:11.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836504 -lt 21474836504 2023-09-30T01:12:11.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:11.361 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672990 2023-09-30T01:12:11.361 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:11.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2023-09-30T01:12:11.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672990 2023-09-30T01:12:11.365 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:11.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672990 2023-09-30T01:12:11.367 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.1 seq 42949672990 2023-09-30T01:12:11.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2023-09-30T01:12:11.367 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:12:11.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672990 -lt 42949672990 2023-09-30T01:12:11.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:11.798 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509460 2023-09-30T01:12:11.798 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:11.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2023-09-30T01:12:11.801 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509460 2023-09-30T01:12:11.801 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:11.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509460 2023-09-30T01:12:11.803 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.2 seq 64424509460 2023-09-30T01:12:11.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2023-09-30T01:12:11.804 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2023-09-30T01:12:12.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509459 -lt 64424509460 2023-09-30T01:12:12.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:12:13.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:12:13.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2023-09-30T01:12:13.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509460 -lt 64424509460 2023-09-30T01:12:13.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:13.653 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 3-85899345930 2023-09-30T01:12:13.653 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:13.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=3 2023-09-30T01:12:13.656 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 3-85899345930 2023-09-30T01:12:13.657 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:13.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=85899345930 2023-09-30T01:12:13.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2023-09-30T01:12:13.659 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.3 seq 85899345930 2023-09-30T01:12:13.659 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 3 2023-09-30T01:12:14.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 85899345930 -lt 85899345930 2023-09-30T01:12:14.050 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:12:14.050 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:14.051 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:14.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2023-09-30T01:12:14.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:12:14.554 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:12:14.554 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:12:14.554 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:12:14.554 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:12:14.554 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:12:14.554 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:12:14.978 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=19 2023-09-30T01:12:14.979 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:12:14.979 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:14.980 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:15.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 19 = 32 2023-09-30T01:12:15.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 19 '!=' -1 2023-09-30T01:12:15.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-09-30T01:12:15.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=19 2023-09-30T01:12:15.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-09-30T01:12:15.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-09-30T01:12:15.588 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-09-30T01:12:15.588 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:12:15.589 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:12:15.589 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:12:15.589 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:12:15.589 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:12:15.589 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:12:15.589 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:12:16.030 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=19 2023-09-30T01:12:16.030 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:12:16.031 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:16.031 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:16.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 19 = 32 2023-09-30T01:12:16.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 19 '!=' 19 2023-09-30T01:12:16.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-09-30T01:12:16.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-09-30T01:12:16.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-09-30T01:12:16.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-09-30T01:12:16.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-09-30T01:12:16.547 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-09-30T01:12:16.547 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-09-30T01:12:17.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-09-30T01:12:17.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-09-30T01:12:17.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2023-09-30T01:12:17.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-09-30T01:12:17.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2023-09-30T01:12:17.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-09-30T01:12:17.279 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:12:17.279 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:12:17.279 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:12:17.279 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:12:17.280 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:12:17.280 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:12:17.280 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:12:17.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2023-09-30T01:12:17.665 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:12:17.666 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:17.666 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:18.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2023-09-30T01:12:18.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:12:18.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:12:18.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2023-09-30T01:12:19.067 INFO:tasks.workunit.client.0.smithi186.stderr:set pool 1 min_size to 3 2023-09-30T01:12:19.079 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2023-09-30T01:12:19.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:12:19.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:12:19.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:12:19.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:12:19.080 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:12:19.081 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:12:19.081 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:12:19.082 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:12:19.082 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:12:19.082 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:12:19.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:12:19.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:12:19.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:12:19.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:12:19.282 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:12:19.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2023-09-30T01:12:19.707 INFO:tasks.workunit.client.0.smithi186.stderr:1 2023-09-30T01:12:19.707 INFO:tasks.workunit.client.0.smithi186.stderr:2 2023-09-30T01:12:19.707 INFO:tasks.workunit.client.0.smithi186.stderr:3' 2023-09-30T01:12:19.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:12:19.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:19.708 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:12:19.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836508 2023-09-30T01:12:19.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836508 2023-09-30T01:12:19.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836508' 2023-09-30T01:12:19.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:19.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-09-30T01:12:20.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672994 2023-09-30T01:12:20.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672994 2023-09-30T01:12:20.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836508 1-42949672994' 2023-09-30T01:12:20.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:20.037 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-09-30T01:12:20.183 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509464 2023-09-30T01:12:20.183 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509464 2023-09-30T01:12:20.183 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836508 1-42949672994 2-64424509464' 2023-09-30T01:12:20.183 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:20.183 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-09-30T01:12:20.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=85899345934 2023-09-30T01:12:20.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 85899345934 2023-09-30T01:12:20.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836508 1-42949672994 2-64424509464 3-85899345934' 2023-09-30T01:12:20.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:20.332 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836508 2023-09-30T01:12:20.332 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:20.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:12:20.335 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836508 2023-09-30T01:12:20.335 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:20.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836508 2023-09-30T01:12:20.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2023-09-30T01:12:20.337 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836508 2023-09-30T01:12:20.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:12:20.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836508 -lt 21474836508 2023-09-30T01:12:20.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:20.791 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672994 2023-09-30T01:12:20.792 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:20.794 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2023-09-30T01:12:20.795 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672994 2023-09-30T01:12:20.795 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:20.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672994 2023-09-30T01:12:20.797 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.1 seq 42949672994 2023-09-30T01:12:20.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2023-09-30T01:12:20.797 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:12:21.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672994 -lt 42949672994 2023-09-30T01:12:21.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:21.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509464 2023-09-30T01:12:21.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:21.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2023-09-30T01:12:21.212 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509464 2023-09-30T01:12:21.212 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:21.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509464 2023-09-30T01:12:21.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2023-09-30T01:12:21.215 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.2 seq 64424509464 2023-09-30T01:12:21.215 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2023-09-30T01:12:21.705 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509464 -lt 64424509464 2023-09-30T01:12:21.705 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:21.706 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 3-85899345934 2023-09-30T01:12:21.706 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:21.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=3 2023-09-30T01:12:21.709 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 3-85899345934 2023-09-30T01:12:21.709 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:21.711 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=85899345934 2023-09-30T01:12:21.711 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2023-09-30T01:12:21.712 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.3 seq 85899345934 2023-09-30T01:12:21.712 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 3 2023-09-30T01:12:22.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 85899345934 -lt 85899345934 2023-09-30T01:12:22.122 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:12:22.122 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:22.122 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:22.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2023-09-30T01:12:22.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:12:22.660 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:12:22.660 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:12:22.660 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:12:22.660 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:12:22.660 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:12:22.660 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:12:23.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2023-09-30T01:12:23.070 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:12:23.070 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:23.070 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:23.587 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2023-09-30T01:12:23.588 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:12:23.588 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:12:23.588 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2023-09-30T01:12:23.975 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:23.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2023-09-30T01:12:24.370 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:24.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2023-09-30T01:12:24.753 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:24.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2023-09-30T01:12:25.142 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:25.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2023-09-30T01:12:25.449 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:25.450 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2023-09-30T01:12:25.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2023-09-30T01:12:25.720 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:25.722 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2023-09-30T01:12:25.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2023-09-30T01:12:25.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2023-09-30T01:12:26.120 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:26.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2023-09-30T01:12:26.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2023-09-30T01:12:26.508 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:26.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2023-09-30T01:12:27.569 INFO:tasks.workunit.client.0.smithi186.stderr:set pool 1 min_size to 2 2023-09-30T01:12:27.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2023-09-30T01:12:27.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:12:27.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:12:27.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:12:27.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:12:27.582 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:12:27.583 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:12:27.583 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:12:27.583 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:12:27.583 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:12:27.583 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:12:27.773 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:12:27.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:12:27.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:12:27.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:12:27.774 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:12:28.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2023-09-30T01:12:28.476 INFO:tasks.workunit.client.0.smithi186.stderr:1 2023-09-30T01:12:28.476 INFO:tasks.workunit.client.0.smithi186.stderr:2 2023-09-30T01:12:28.476 INFO:tasks.workunit.client.0.smithi186.stderr:3' 2023-09-30T01:12:28.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:12:28.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:28.476 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:12:28.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836512 2023-09-30T01:12:28.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836512 2023-09-30T01:12:28.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836512' 2023-09-30T01:12:28.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:28.629 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-09-30T01:12:28.773 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672997 2023-09-30T01:12:28.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672997 2023-09-30T01:12:28.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836512 1-42949672997' 2023-09-30T01:12:28.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:28.774 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-09-30T01:12:28.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509467 2023-09-30T01:12:28.920 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509467 2023-09-30T01:12:28.920 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836512 1-42949672997 2-64424509467' 2023-09-30T01:12:28.920 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:28.920 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-09-30T01:12:29.055 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=85899345938 2023-09-30T01:12:29.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 85899345938 2023-09-30T01:12:29.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836512 1-42949672997 2-64424509467 3-85899345938' 2023-09-30T01:12:29.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:29.056 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836512 2023-09-30T01:12:29.057 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:29.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:12:29.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836512 2023-09-30T01:12:29.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:29.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836512 2023-09-30T01:12:29.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2023-09-30T01:12:29.062 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836512 2023-09-30T01:12:29.062 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:12:29.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836510 -lt 21474836512 2023-09-30T01:12:29.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:12:30.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:12:30.477 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:12:30.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836512 -lt 21474836512 2023-09-30T01:12:30.870 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:30.870 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672997 2023-09-30T01:12:30.870 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:30.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2023-09-30T01:12:30.873 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672997 2023-09-30T01:12:30.873 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:30.875 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.1 seq 42949672997 2023-09-30T01:12:30.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672997 2023-09-30T01:12:30.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2023-09-30T01:12:30.876 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:12:31.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672998 -lt 42949672997 2023-09-30T01:12:31.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:31.298 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:31.298 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509467 2023-09-30T01:12:31.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2023-09-30T01:12:31.301 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509467 2023-09-30T01:12:31.301 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:31.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509467 2023-09-30T01:12:31.304 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.2 seq 64424509467 2023-09-30T01:12:31.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2023-09-30T01:12:31.304 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2023-09-30T01:12:31.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509468 -lt 64424509467 2023-09-30T01:12:31.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:31.738 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:31.739 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 3-85899345938 2023-09-30T01:12:31.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=3 2023-09-30T01:12:31.741 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 3-85899345938 2023-09-30T01:12:31.741 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:31.742 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=85899345938 2023-09-30T01:12:31.743 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.3 seq 85899345938 2023-09-30T01:12:31.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.3 seq 85899345938' 2023-09-30T01:12:31.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 3 2023-09-30T01:12:32.140 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 85899345938 -lt 85899345938 2023-09-30T01:12:32.140 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:12:32.141 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:32.141 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:32.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2023-09-30T01:12:32.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:12:32.681 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:12:32.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:12:32.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:12:32.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:12:32.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:12:32.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:12:33.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=25 2023-09-30T01:12:33.065 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:12:33.065 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:33.066 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:33.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 25 = 32 2023-09-30T01:12:33.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 25 '!=' -1 2023-09-30T01:12:33.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-09-30T01:12:33.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=25 2023-09-30T01:12:33.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-09-30T01:12:33.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-09-30T01:12:33.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-09-30T01:12:33.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:12:33.687 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:12:33.687 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:12:33.687 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:12:33.687 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:12:33.688 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:12:33.688 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:12:34.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=25 2023-09-30T01:12:34.085 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:12:34.086 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:34.086 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:34.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 25 = 32 2023-09-30T01:12:34.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 25 '!=' 25 2023-09-30T01:12:34.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-09-30T01:12:34.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-09-30T01:12:34.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-09-30T01:12:34.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-09-30T01:12:34.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-09-30T01:12:34.599 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-09-30T01:12:34.599 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-09-30T01:12:35.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-09-30T01:12:35.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-09-30T01:12:35.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2023-09-30T01:12:35.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-09-30T01:12:35.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2023-09-30T01:12:35.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-09-30T01:12:35.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:12:35.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:12:35.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:12:35.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:12:35.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:12:35.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:12:35.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:12:35.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2023-09-30T01:12:35.716 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:12:35.716 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:35.716 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:36.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2023-09-30T01:12:36.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:12:36.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:12:36.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2023-09-30T01:12:36.573 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:36.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2023-09-30T01:12:37.022 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:37.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2023-09-30T01:12:37.319 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:37.321 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2023-09-30T01:12:37.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2023-09-30T01:12:37.605 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:37.606 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2023-09-30T01:12:37.610 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2023-09-30T01:12:37.611 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2023-09-30T01:12:37.986 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:37.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2023-09-30T01:12:37.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2023-09-30T01:12:38.381 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:38.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2023-09-30T01:12:38.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2023-09-30T01:12:38.766 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:38.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2023-09-30T01:12:38.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2023-09-30T01:12:39.163 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:39.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2023-09-30T01:12:39.164 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:12:39.164 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:12:39.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:12:39.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:12:39.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:12:39.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:12:39.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2023-09-30T01:12:40.634 INFO:tasks.workunit.client.0.smithi186.stderr:osd.0 is already down. 2023-09-30T01:12:40.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2023-09-30T01:12:40.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2023-09-30T01:12:40.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2023-09-30T01:12:40.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2023-09-30T01:12:40.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:12:40.649 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2023-09-30T01:12:40.649 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:12:40.650 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:12:40.650 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:12:40.650 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:12:40.650 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:12:40.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2023-09-30T01:12:40.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2023-09-30T01:12:40.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2023-09-30T01:12:40.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:12:40.854 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:12:41.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2023-09-30T01:12:41.278 INFO:tasks.workunit.client.0.smithi186.stderr:1 2023-09-30T01:12:41.278 INFO:tasks.workunit.client.0.smithi186.stderr:2 2023-09-30T01:12:41.278 INFO:tasks.workunit.client.0.smithi186.stderr:3' 2023-09-30T01:12:41.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:12:41.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:41.278 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:12:41.457 INFO:tasks.workunit.client.0.smithi186.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-09-30T01:12:41.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq= 2023-09-30T01:12:41.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z '' 2023-09-30T01:12:41.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: flush_pg_stats: continue 2023-09-30T01:12:41.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:41.460 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-09-30T01:12:41.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949673002 2023-09-30T01:12:41.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949673002 2023-09-30T01:12:41.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 1-42949673002' 2023-09-30T01:12:41.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:41.688 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-09-30T01:12:41.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=64424509472 2023-09-30T01:12:41.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 64424509472 2023-09-30T01:12:41.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 1-42949673002 2-64424509472' 2023-09-30T01:12:41.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:12:41.840 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-09-30T01:12:41.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=85899345942 2023-09-30T01:12:41.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 85899345942 2023-09-30T01:12:41.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 1-42949673002 2-64424509472 3-85899345942' 2023-09-30T01:12:41.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:41.983 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949673002 2023-09-30T01:12:41.983 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:41.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2023-09-30T01:12:41.984 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949673002 2023-09-30T01:12:41.985 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:41.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949673002 2023-09-30T01:12:41.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2023-09-30T01:12:41.987 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.1 seq 42949673002 2023-09-30T01:12:41.987 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:12:42.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949673000 -lt 42949673002 2023-09-30T01:12:42.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:12:43.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:12:43.384 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:12:43.826 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949673002 -lt 42949673002 2023-09-30T01:12:43.826 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:43.827 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 2-64424509472 2023-09-30T01:12:43.827 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:43.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=2 2023-09-30T01:12:43.829 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 2-64424509472 2023-09-30T01:12:43.830 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:43.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=64424509472 2023-09-30T01:12:43.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2023-09-30T01:12:43.832 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.2 seq 64424509472 2023-09-30T01:12:43.832 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 2 2023-09-30T01:12:44.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 64424509472 -lt 64424509472 2023-09-30T01:12:44.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:12:44.311 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 3-85899345942 2023-09-30T01:12:44.311 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:12:44.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=3 2023-09-30T01:12:44.314 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 3-85899345942 2023-09-30T01:12:44.314 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:12:44.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=85899345942 2023-09-30T01:12:44.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2023-09-30T01:12:44.316 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.3 seq 85899345942 2023-09-30T01:12:44.317 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 3 2023-09-30T01:12:44.705 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 85899345943 -lt 85899345942 2023-09-30T01:12:44.706 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2023-09-30T01:12:44.706 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:44.706 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:45.223 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 == 0 2023-09-30T01:12:45.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2023-09-30T01:12:45.224 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2023-09-30T01:12:45.225 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2023-09-30T01:12:45.225 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2023-09-30T01:12:45.225 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2023-09-30T01:12:45.225 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2023-09-30T01:12:45.225 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2023-09-30T01:12:45.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=32 2023-09-30T01:12:45.646 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2023-09-30T01:12:45.647 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:12:45.647 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:12:46.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 32 = 32 2023-09-30T01:12:46.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2023-09-30T01:12:46.156 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2023-09-30T01:12:46.156 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2023-09-30T01:12:46.616 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2023-09-30T01:12:46.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2023-09-30T01:12:47.011 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:47.023 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2023-09-30T01:12:47.283 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:47.284 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2023-09-30T01:12:47.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2023-09-30T01:12:47.554 INFO:tasks.workunit.client.0.smithi186.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2023-09-30T01:12:47.555 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2023-09-30T01:12:47.558 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2023-09-30T01:12:47.559 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:12:47.559 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:12:47.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:12:47.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:12:47.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:12:47.676 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:12:47.677 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2023-09-30T01:12:47.677 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:12:47.677 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:12:47.677 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:12:47.677 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:12:47.677 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:12:47.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:12:47.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:12:47.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:12:47.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:12:47.692 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:12:47.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:12:47.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:12:47.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:12:47.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:12:47.695 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:12:47.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:12:47.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:12:47.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:12:47.697 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:12:47.697 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:12:47.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:12:47.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:12:47.721 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:12:47.721 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:47.721 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:47.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:12:47.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:12:47.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:12:47.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2023-09-30T01:12:47.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2023-09-30T01:12:47.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2023-09-30T01:12:47.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2023-09-30T01:12:47.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:12:47.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:12:47.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:12:47.724 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:12:47.725 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:12:47.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:12:47.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:12:47.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:12:47.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:12:47.728 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:12:47.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:12:47.730 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:12:47.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:12:47.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:12:47.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:12:47.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:12:47.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:12:47.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:12:47.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:12:47.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:12:47.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:12:47.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:12:47.738 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:12:47.739 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:47.739 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:47.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:12:47.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:12:47.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:12:47.741 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2023-09-30T01:12:47.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:12:47.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:47.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:47.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53393 2023-09-30T01:12:47.745 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:12:47.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:12:47.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:12:47.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2023-09-30T01:12:47.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2023-09-30T01:12:47.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2023-09-30T01:12:47.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150 ' 2023-09-30T01:12:47.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2023-09-30T01:12:47.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:12:47.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:12:47.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:12:47.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:12:47.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2023-09-30T01:12:47.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2023-09-30T01:12:48.114 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:12:48.114 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:12:48.114 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:12:48.115 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:12:48.115 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:48.115 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:48.116 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:12:48.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2023-09-30T01:12:48.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:12:48.430 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:12:48.431 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:12:48.431 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:12:48.431 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:12:48.431 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:12:48.432 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:12:48.432 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:12:48.432 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:12:48.433 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:12:48.433 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:48.433 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:48.433 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:12:48.434 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:12:48.434 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get fsid 2023-09-30T01:12:48.526 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:12:48.526 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:12:48.526 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:12:48.526 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:12:48.527 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:12:48.528 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:12:48.528 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:12:48.528 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:12:48.528 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:12:48.528 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:48.528 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:48.528 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:12:48.529 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:12:48.529 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get mon_host 2023-09-30T01:12:48.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2023-09-30T01:12:49.437 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a) after stopping [dne] 2023-09-30T01:12:49.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2023-09-30T01:12:49.662 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2023-09-30T01:12:49.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2023-09-30T01:12:49.830 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2023-09-30T01:12:49.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2023-09-30T01:12:50.056 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: removing mon.a would break quorum 2023-09-30T01:12:50.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2023-09-30T01:12:50.280 INFO:tasks.workunit.client.0.smithi186.stderr:mon.dne does not exist 2023-09-30T01:12:50.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2023-09-30T01:12:50.289 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:12:50.289 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:12:50.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:12:50.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:12:50.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:12:50.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:12:50.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2023-09-30T01:12:50.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:12:50.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:12:50.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:12:50.404 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:12:50.404 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:12:50.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:12:50.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:12:50.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:12:50.410 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:12:50.411 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:12:50.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:12:50.413 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:12:50.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:12:50.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:12:50.416 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:12:50.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:12:50.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:12:50.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:12:50.419 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:12:50.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:12:50.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:12:50.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:12:50.425 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:12:50.425 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:50.425 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:50.425 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:12:50.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:12:50.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:12:50.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2023-09-30T01:12:50.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2023-09-30T01:12:50.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2023-09-30T01:12:50.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2023-09-30T01:12:50.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:12:50.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:12:50.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:12:50.429 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:12:50.429 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:12:50.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:12:50.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:12:50.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:12:50.432 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:12:50.433 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:12:50.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:12:50.435 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:12:50.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:12:50.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:12:50.437 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:12:50.439 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:12:50.439 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:12:50.439 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:12:50.440 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:12:50.441 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:12:50.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:12:50.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:12:50.446 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:12:50.446 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:50.446 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:50.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:12:50.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:12:50.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:12:50.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2023-09-30T01:12:50.450 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:12:50.450 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:50.451 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:50.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53393 2023-09-30T01:12:50.453 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:12:50.455 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:12:50.455 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:12:50.455 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2023-09-30T01:12:50.455 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2023-09-30T01:12:50.455 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2023-09-30T01:12:50.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2023-09-30T01:12:50.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2023-09-30T01:12:50.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:12:50.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:12:50.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:12:50.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:12:50.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2023-09-30T01:12:50.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2023-09-30T01:12:50.832 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:12:50.833 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:12:50.833 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:12:50.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:12:50.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:50.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:50.835 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:12:50.835 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2023-09-30T01:12:51.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:12:51.117 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:12:51.117 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:12:51.117 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:12:51.117 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:12:51.118 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:12:51.119 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:12:51.119 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:12:51.119 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:12:51.120 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:12:51.120 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:51.120 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:51.120 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:12:51.121 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:12:51.121 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get fsid 2023-09-30T01:12:51.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:12:51.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:12:51.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:12:51.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:12:51.211 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:12:51.211 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:12:51.211 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:12:51.211 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:12:51.212 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:12:51.212 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:51.212 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:51.213 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:12:51.213 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:12:51.213 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get mon_host 2023-09-30T01:12:51.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2023-09-30T01:12:51.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:12:51.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:12:51.298 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2023-09-30T01:12:51.298 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:12:51.298 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2023-09-30T01:12:51.298 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2023-09-30T01:12:51.722 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:12:51.722 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:12:51.723 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:12:51.723 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:12:51.724 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:51.724 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:51.724 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:12:51.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2023-09-30T01:12:52.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:12:52.004 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2023-09-30T01:12:52.004 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:12:52.004 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2023-09-30T01:12:52.004 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:12:52.005 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:12:52.005 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2023-09-30T01:12:52.005 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2023-09-30T01:12:52.005 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2023-09-30T01:12:52.006 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:12:52.006 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:52.006 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:52.007 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.b.asok 2023-09-30T01:12:52.008 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:12:52.008 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.b.asok config get fsid 2023-09-30T01:12:52.099 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2023-09-30T01:12:52.099 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:12:52.099 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2023-09-30T01:12:52.099 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:12:52.100 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:12:52.100 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2023-09-30T01:12:52.100 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2023-09-30T01:12:52.100 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2023-09-30T01:12:52.100 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:12:52.100 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:12:52.100 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:12:52.101 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.b.asok 2023-09-30T01:12:52.101 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:12:52.101 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.b.asok config get mon_host 2023-09-30T01:12:52.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2023-09-30T01:12:58.401 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,b) after stopping [dne] 2023-09-30T01:12:58.410 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2023-09-30T01:12:58.589 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2023-09-30T01:12:58.591 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2023-09-30T01:12:58.763 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2023-09-30T01:12:58.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2023-09-30T01:12:58.957 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2023-09-30T01:12:58.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2023-09-30T01:12:59.293 INFO:tasks.workunit.client.0.smithi186.stderr:adding another mon that is not yet online will not break quorum 2023-09-30T01:12:59.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2023-09-30T01:12:59.599 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.a 2023-09-30T01:12:59.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2023-09-30T01:12:59.917 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.b 2023-09-30T01:12:59.926 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2023-09-30T01:13:00.180 INFO:tasks.workunit.client.0.smithi186.stderr:mon.dne does not exist 2023-09-30T01:13:00.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2023-09-30T01:13:00.189 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:13:00.189 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:13:00.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:13:00.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:13:00.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:13:00.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:13:00.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2023-09-30T01:13:00.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:13:00.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:13:00.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:13:00.303 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:13:00.303 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:13:00.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:13:00.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:13:00.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:13:00.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:13:00.314 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:13:00.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:13:00.315 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:13:00.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:13:00.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:13:00.317 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:13:00.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:13:00.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:13:00.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:13:00.319 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:13:00.320 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:13:00.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:13:00.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:13:00.325 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:13:00.325 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:00.325 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:00.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:13:00.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:13:00.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:13:00.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2023-09-30T01:13:00.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2023-09-30T01:13:00.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2023-09-30T01:13:00.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2023-09-30T01:13:00.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:13:00.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:13:00.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:13:00.329 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:13:00.329 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:13:00.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:13:00.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:13:00.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:13:00.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:13:00.334 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:13:00.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:13:00.334 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:13:00.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:13:00.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:13:00.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:13:00.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:13:00.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:13:00.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:13:00.338 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:13:00.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:13:00.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:13:00.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:13:00.344 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:13:00.344 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:00.344 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:00.344 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:13:00.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:13:00.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:13:00.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2023-09-30T01:13:00.348 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:13:00.348 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:00.348 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:00.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53393 2023-09-30T01:13:00.351 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:13:00.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:13:00.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:13:00.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2023-09-30T01:13:00.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2023-09-30T01:13:00.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2023-09-30T01:13:00.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2023-09-30T01:13:00.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2023-09-30T01:13:00.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:13:00.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:13:00.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:13:00.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:13:00.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2023-09-30T01:13:00.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2023-09-30T01:13:00.721 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:13:00.722 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:13:00.722 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:13:00.722 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:13:00.722 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:00.723 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:00.723 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:13:00.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2023-09-30T01:13:01.036 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:13:01.038 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:13:01.038 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:01.038 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:13:01.038 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:13:01.038 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:13:01.039 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:13:01.039 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:13:01.039 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:13:01.040 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:01.040 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:01.040 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:01.040 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:13:01.041 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:01.041 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get fsid 2023-09-30T01:13:01.133 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:13:01.133 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:01.133 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:13:01.133 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:13:01.134 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:13:01.134 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:13:01.135 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:13:01.135 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:13:01.135 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:01.135 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:01.135 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:01.135 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:13:01.136 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:01.136 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get mon_host 2023-09-30T01:13:01.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2023-09-30T01:13:01.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:13:01.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:13:01.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2023-09-30T01:13:01.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:13:01.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2023-09-30T01:13:01.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2023-09-30T01:13:01.592 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:13:01.592 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:13:01.592 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:13:01.593 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:13:01.593 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:01.593 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:01.593 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:13:01.593 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2023-09-30T01:13:01.874 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:13:01.876 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2023-09-30T01:13:01.876 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:01.876 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2023-09-30T01:13:01.876 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:13:01.876 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:13:01.877 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2023-09-30T01:13:01.877 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2023-09-30T01:13:01.877 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2023-09-30T01:13:01.878 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:01.878 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:01.878 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:01.879 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.b.asok 2023-09-30T01:13:01.879 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:01.879 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.b.asok config get fsid 2023-09-30T01:13:01.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2023-09-30T01:13:01.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:01.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2023-09-30T01:13:01.973 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:13:01.974 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:13:01.974 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2023-09-30T01:13:01.974 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2023-09-30T01:13:01.974 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2023-09-30T01:13:01.974 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:01.974 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:01.974 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:01.975 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.b.asok 2023-09-30T01:13:01.975 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:01.975 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.b.asok config get mon_host 2023-09-30T01:13:02.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2023-09-30T01:13:02.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:13:02.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:13:02.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2023-09-30T01:13:02.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:13:02.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2023-09-30T01:13:02.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2023-09-30T01:13:02.454 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:13:02.454 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:13:02.454 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:13:02.455 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:13:02.455 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:02.455 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:02.456 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:13:02.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2023-09-30T01:13:02.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:13:02.747 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2023-09-30T01:13:02.747 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:02.747 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2023-09-30T01:13:02.747 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:13:02.748 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:13:02.749 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2023-09-30T01:13:02.749 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2023-09-30T01:13:02.749 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2023-09-30T01:13:02.749 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:02.750 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:02.750 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:02.750 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.c.asok 2023-09-30T01:13:02.751 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:02.751 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.c.asok config get fsid 2023-09-30T01:13:02.838 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2023-09-30T01:13:02.838 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:02.838 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2023-09-30T01:13:02.839 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:13:02.839 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2023-09-30T01:13:02.840 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:13:02.840 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2023-09-30T01:13:02.840 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2023-09-30T01:13:02.840 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:02.840 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:02.840 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:02.840 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.c.asok 2023-09-30T01:13:02.841 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:02.841 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.c.asok config get mon_host 2023-09-30T01:13:02.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2023-09-30T01:13:02.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2023-09-30T01:13:02.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2023-09-30T01:13:02.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2023-09-30T01:13:02.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2023-09-30T01:13:02.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2023-09-30T01:13:02.924 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2023-09-30T01:13:02.924 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1696036442 2023-09-30T01:13:02.925 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:02.926 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036382 -lt 1696036442 ]] 2023-09-30T01:13:02.926 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:02.927 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:09.149 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:09.150 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:09.150 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:09.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:09.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2023-09-30T01:13:09.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2023-09-30T01:13:09.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2023-09-30T01:13:09.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2023-09-30T01:13:09.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2023-09-30T01:13:09.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2023-09-30T01:13:12.412 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,b,c) after stopping [dne] 2023-09-30T01:13:12.420 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2023-09-30T01:13:12.729 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (b,c) after stopping [a] 2023-09-30T01:13:12.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2023-09-30T01:13:13.005 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,c) after stopping [b] 2023-09-30T01:13:13.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2023-09-30T01:13:13.363 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,b) after stopping [c] 2023-09-30T01:13:13.371 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2023-09-30T01:13:13.620 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2023-09-30T01:13:13.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2023-09-30T01:13:13.887 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2023-09-30T01:13:13.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2023-09-30T01:13:14.122 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2023-09-30T01:13:14.125 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2023-09-30T01:13:14.414 INFO:tasks.workunit.client.0.smithi186.stderr:adding another mon that is not yet online will not break quorum 2023-09-30T01:13:14.423 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2023-09-30T01:13:14.844 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.a 2023-09-30T01:13:14.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2023-09-30T01:13:15.118 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.b 2023-09-30T01:13:15.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2023-09-30T01:13:15.524 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.c 2023-09-30T01:13:15.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2023-09-30T01:13:15.539 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:13:15.539 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:13:15.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:13:15.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:13:15.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:13:15.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:13:15.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2023-09-30T01:13:15.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2023-09-30T01:13:15.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2023-09-30T01:13:15.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2023-09-30T01:13:15.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2023-09-30T01:13:15.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2023-09-30T01:13:15.652 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2023-09-30T01:13:15.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1696036455 2023-09-30T01:13:15.653 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:15.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036395 -lt 1696036455 ]] 2023-09-30T01:13:15.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2023-09-30T01:13:15.656 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:18.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:18.890 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:18.891 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:18.891 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2023-09-30T01:13:18.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:18.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:18.932 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:18.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036398 -lt 1696036455 ]] 2023-09-30T01:13:18.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2023-09-30T01:13:18.934 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:19.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:19.146 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:19.149 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:19.149 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2023-09-30T01:13:19.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:19.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:19.189 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:19.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036399 -lt 1696036455 ]] 2023-09-30T01:13:19.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2023-09-30T01:13:19.191 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:19.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:19.412 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:19.413 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2023-09-30T01:13:19.413 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:19.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:19.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:19.450 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:19.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036399 -lt 1696036455 ]] 2023-09-30T01:13:19.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2023-09-30T01:13:19.452 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:22.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:22.680 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:22.681 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2023-09-30T01:13:22.681 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:22.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:22.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:22.724 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:22.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036402 -lt 1696036455 ]] 2023-09-30T01:13:22.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2023-09-30T01:13:22.726 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:31.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:31.955 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:31.956 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:00.375493Z","created":"2023-09-30T01:13:00.375493Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:31.956 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2023-09-30T01:13:31.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2023-09-30T01:13:31.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2023-09-30T01:13:31.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2023-09-30T01:13:31.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2023-09-30T01:13:31.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2023-09-30T01:13:31.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2023-09-30T01:13:32.217 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2023-09-30T01:13:32.219 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2023-09-30T01:13:32.526 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,c) after stopping [b] 2023-09-30T01:13:32.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2023-09-30T01:13:32.724 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2023-09-30T01:13:32.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2023-09-30T01:13:32.891 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2023-09-30T01:13:32.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2023-09-30T01:13:33.181 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: removing mon.a would break quorum 2023-09-30T01:13:33.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2023-09-30T01:13:33.509 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.b 2023-09-30T01:13:33.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2023-09-30T01:13:33.820 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: removing mon.c would break quorum 2023-09-30T01:13:33.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2023-09-30T01:13:33.824 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:13:33.824 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:13:33.824 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:13:33.824 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:13:33.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:13:33.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:13:33.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2023-09-30T01:13:33.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:13:33.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:13:33.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:13:33.939 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:13:33.939 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:13:33.939 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:13:33.939 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:13:33.939 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:13:33.950 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:13:33.950 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:13:33.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:13:33.952 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:13:33.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:13:33.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:13:33.955 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:13:33.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:13:33.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:13:33.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:13:33.957 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:13:33.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:13:33.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:13:33.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:13:33.968 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:13:33.968 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:33.968 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:33.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:13:33.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:13:33.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:13:33.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2023-09-30T01:13:33.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2023-09-30T01:13:33.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2023-09-30T01:13:33.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2023-09-30T01:13:33.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:13:33.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:13:33.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:13:33.971 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:13:33.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:13:33.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:13:33.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:13:33.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:13:33.975 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:13:33.976 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:13:33.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:13:33.978 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:13:33.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:13:33.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:13:33.981 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:13:33.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:13:33.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:13:33.984 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:13:33.984 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:13:33.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:13:33.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:13:33.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:13:33.990 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:13:33.990 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:33.990 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:33.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:13:33.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:13:33.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:13:33.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2023-09-30T01:13:33.994 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:13:33.994 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:33.995 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:33.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53393 2023-09-30T01:13:33.997 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:13:34.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:13:34.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:13:34.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2023-09-30T01:13:34.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2023-09-30T01:13:34.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2023-09-30T01:13:34.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2023-09-30T01:13:34.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2023-09-30T01:13:34.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:13:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:13:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:13:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:13:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2023-09-30T01:13:34.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2023-09-30T01:13:34.359 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:13:34.359 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:13:34.359 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:13:34.360 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:13:34.360 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:34.360 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:34.360 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:13:34.361 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2023-09-30T01:13:34.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:13:34.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:13:34.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:34.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:13:34.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:13:34.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:13:34.635 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:13:34.635 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:13:34.635 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:13:34.636 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:34.636 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:34.636 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:34.636 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:13:34.637 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:34.637 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get fsid 2023-09-30T01:13:34.724 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:13:34.724 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:34.724 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:13:34.724 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:13:34.725 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:13:34.725 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:13:34.725 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:13:34.725 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:13:34.725 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:34.726 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:34.726 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:34.726 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:13:34.726 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:34.726 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get mon_host 2023-09-30T01:13:34.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2023-09-30T01:13:34.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:13:34.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:13:34.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2023-09-30T01:13:34.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:13:34.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2023-09-30T01:13:34.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2023-09-30T01:13:35.196 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:13:35.197 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:13:35.197 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:13:35.197 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:13:35.197 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:35.197 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:35.198 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:13:35.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2023-09-30T01:13:35.471 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:13:35.473 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2023-09-30T01:13:35.473 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:35.473 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2023-09-30T01:13:35.473 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:13:35.473 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:13:35.474 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2023-09-30T01:13:35.474 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2023-09-30T01:13:35.474 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2023-09-30T01:13:35.475 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:35.475 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:35.475 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:35.476 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.b.asok 2023-09-30T01:13:35.476 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:35.476 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.b.asok config get fsid 2023-09-30T01:13:35.569 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2023-09-30T01:13:35.569 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:35.569 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2023-09-30T01:13:35.569 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:13:35.570 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2023-09-30T01:13:35.570 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2023-09-30T01:13:35.570 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:13:35.570 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2023-09-30T01:13:35.570 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:35.570 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:35.570 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:35.571 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.b.asok 2023-09-30T01:13:35.571 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:35.571 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.b.asok config get mon_host 2023-09-30T01:13:35.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2023-09-30T01:13:35.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:13:35.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:13:35.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2023-09-30T01:13:35.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:13:35.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2023-09-30T01:13:35.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2023-09-30T01:13:36.018 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:13:36.019 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:13:36.019 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:13:36.019 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:13:36.019 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:36.019 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:36.019 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:13:36.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2023-09-30T01:13:36.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:13:36.334 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2023-09-30T01:13:36.334 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:36.334 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2023-09-30T01:13:36.334 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:13:36.335 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:13:36.335 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2023-09-30T01:13:36.335 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2023-09-30T01:13:36.335 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2023-09-30T01:13:36.336 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:36.336 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:36.336 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:36.336 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.c.asok 2023-09-30T01:13:36.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:36.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.c.asok config get fsid 2023-09-30T01:13:36.428 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2023-09-30T01:13:36.429 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:36.429 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2023-09-30T01:13:36.429 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:13:36.429 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:13:36.430 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2023-09-30T01:13:36.430 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2023-09-30T01:13:36.430 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2023-09-30T01:13:36.430 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:36.430 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:36.430 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:36.431 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.c.asok 2023-09-30T01:13:36.431 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:36.431 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.c.asok config get mon_host 2023-09-30T01:13:36.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2023-09-30T01:13:36.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:13:36.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:13:36.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2023-09-30T01:13:36.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:13:36.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2023-09-30T01:13:36.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2023-09-30T01:13:36.887 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:13:36.887 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:13:36.887 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:13:36.888 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:13:36.888 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:36.888 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:36.888 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:13:36.888 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2023-09-30T01:13:37.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:13:37.160 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2023-09-30T01:13:37.160 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:37.160 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2023-09-30T01:13:37.160 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:13:37.160 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2023-09-30T01:13:37.161 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2023-09-30T01:13:37.161 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2023-09-30T01:13:37.161 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:13:37.161 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:37.161 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:37.162 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:37.162 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.d.asok 2023-09-30T01:13:37.162 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:37.162 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.d.asok config get fsid 2023-09-30T01:13:37.247 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2023-09-30T01:13:37.247 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:13:37.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2023-09-30T01:13:37.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:13:37.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:13:37.248 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2023-09-30T01:13:37.248 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2023-09-30T01:13:37.248 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2023-09-30T01:13:37.249 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:13:37.249 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:13:37.249 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:13:37.250 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.d.asok 2023-09-30T01:13:37.250 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:13:37.250 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.d.asok config get mon_host 2023-09-30T01:13:37.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2023-09-30T01:13:37.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2023-09-30T01:13:37.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2023-09-30T01:13:37.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2023-09-30T01:13:37.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2023-09-30T01:13:37.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2023-09-30T01:13:37.338 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2023-09-30T01:13:37.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1696036477 2023-09-30T01:13:37.339 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:37.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036417 -lt 1696036477 ]] 2023-09-30T01:13:37.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:13:37.341 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:49.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:49.949 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:49.950 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:13:49.951 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:49.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2023-09-30T01:13:49.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2023-09-30T01:13:49.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2023-09-30T01:13:49.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2023-09-30T01:13:49.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2023-09-30T01:13:49.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2023-09-30T01:13:50.276 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,b,c,d) after stopping [dne] 2023-09-30T01:13:50.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2023-09-30T01:13:50.617 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (b,c,d) after stopping [a] 2023-09-30T01:13:50.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2023-09-30T01:13:50.883 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,c,d) after stopping [b] 2023-09-30T01:13:50.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2023-09-30T01:13:51.149 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,b,d) after stopping [c] 2023-09-30T01:13:51.159 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2023-09-30T01:13:51.594 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,b,c) after stopping [d] 2023-09-30T01:13:51.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2023-09-30T01:13:51.910 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2023-09-30T01:13:51.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2023-09-30T01:13:52.127 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2023-09-30T01:13:52.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2023-09-30T01:13:52.470 INFO:tasks.workunit.client.0.smithi186.stderr:adding another mon that is not yet online will not break quorum 2023-09-30T01:13:52.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2023-09-30T01:13:52.846 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.a 2023-09-30T01:13:52.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2023-09-30T01:13:53.191 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.b 2023-09-30T01:13:53.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2023-09-30T01:13:53.562 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.c 2023-09-30T01:13:53.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2023-09-30T01:13:53.571 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:13:53.572 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:13:53.572 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:13:53.572 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:13:53.572 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:13:53.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:13:53.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2023-09-30T01:13:53.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2023-09-30T01:13:53.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2023-09-30T01:13:53.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2023-09-30T01:13:53.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2023-09-30T01:13:53.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2023-09-30T01:13:53.684 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2023-09-30T01:13:53.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1696036493 2023-09-30T01:13:53.686 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:53.687 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036433 -lt 1696036493 ]] 2023-09-30T01:13:53.687 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:53.688 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:53.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:53.912 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:53.913 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:53.913 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:53.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:53.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:53.956 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:53.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036433 -lt 1696036493 ]] 2023-09-30T01:13:53.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:53.958 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:54.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:54.173 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:54.174 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:54.174 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:54.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:54.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:54.216 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:54.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036434 -lt 1696036493 ]] 2023-09-30T01:13:54.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:54.218 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:54.436 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:54.436 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:54.437 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:54.438 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:54.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:54.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:54.480 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:54.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036434 -lt 1696036493 ]] 2023-09-30T01:13:54.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:54.481 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:54.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:54.711 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:54.712 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:54.712 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:54.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:54.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:54.751 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:54.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036434 -lt 1696036493 ]] 2023-09-30T01:13:54.752 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:54.752 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:54.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:54.974 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:54.974 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:54.975 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:55.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:55.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:55.016 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:55.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036435 -lt 1696036493 ]] 2023-09-30T01:13:55.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:55.019 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:55.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:55.238 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:55.239 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:55.239 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:55.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:55.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:55.281 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:55.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036435 -lt 1696036493 ]] 2023-09-30T01:13:55.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:55.282 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:55.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:55.498 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:55.498 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:55.498 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:55.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:55.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:55.540 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:55.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036435 -lt 1696036493 ]] 2023-09-30T01:13:55.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:55.542 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:55.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:55.765 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:55.767 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:55.767 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:55.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:55.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:55.808 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:55.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036435 -lt 1696036493 ]] 2023-09-30T01:13:55.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:55.810 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:56.029 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:56.029 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:56.030 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:56.030 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:56.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:56.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:56.073 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:56.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036436 -lt 1696036493 ]] 2023-09-30T01:13:56.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:56.074 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:59.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:59.297 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:59.298 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:59.298 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:59.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:59.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:59.343 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:59.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036439 -lt 1696036493 ]] 2023-09-30T01:13:59.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:59.346 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:13:59.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:13:59.582 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:59.583 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:13:59.583 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:13:59.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:13:59.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:13:59.621 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:13:59.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036439 -lt 1696036493 ]] 2023-09-30T01:13:59.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:13:59.623 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:02.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:02.849 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:02.850 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:14:02.850 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:02.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:02.892 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:02.892 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:02.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036442 -lt 1696036493 ]] 2023-09-30T01:14:02.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:14:02.894 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:03.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:03.117 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:03.118 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:03.118 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:14:03.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:03.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:03.162 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:03.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036443 -lt 1696036493 ]] 2023-09-30T01:14:03.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:14:03.164 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:13.856 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:13.857 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:13.857 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:14:13.858 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:13:34.022628Z","created":"2023-09-30T01:13:34.022628Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:13.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2023-09-30T01:14:13.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2023-09-30T01:14:13.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2023-09-30T01:14:13.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2023-09-30T01:14:13.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2023-09-30T01:14:13.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2023-09-30T01:14:17.112 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (b,c,d) after stopping [a] 2023-09-30T01:14:17.120 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2023-09-30T01:14:17.310 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2023-09-30T01:14:17.313 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2023-09-30T01:14:17.562 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2023-09-30T01:14:17.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2023-09-30T01:14:20.743 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2023-09-30T01:14:20.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2023-09-30T01:14:21.040 INFO:tasks.workunit.client.0.smithi186.stderr:adding another mon that is not yet online will not break quorum 2023-09-30T01:14:21.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2023-09-30T01:14:21.366 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.a 2023-09-30T01:14:21.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2023-09-30T01:14:21.642 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.b 2023-09-30T01:14:21.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2023-09-30T01:14:21.937 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.c 2023-09-30T01:14:21.945 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2023-09-30T01:14:22.235 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.d 2023-09-30T01:14:22.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2023-09-30T01:14:22.244 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:14:22.244 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:14:22.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:14:22.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:14:22.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:14:22.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:14:22.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2023-09-30T01:14:22.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:14:22.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:14:22.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:14:22.359 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:14:22.359 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:14:22.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:14:22.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:14:22.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:14:22.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:14:22.372 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:14:22.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:14:22.374 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:14:22.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:14:22.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:14:22.377 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:14:22.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:14:22.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:14:22.379 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:14:22.379 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:14:22.380 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:14:22.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:14:22.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:14:22.389 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:14:22.389 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:22.389 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:22.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:14:22.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:14:22.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:14:22.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2023-09-30T01:14:22.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2023-09-30T01:14:22.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2023-09-30T01:14:22.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2023-09-30T01:14:22.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:14:22.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:14:22.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:14:22.392 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:14:22.392 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:14:22.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:14:22.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:14:22.393 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:14:22.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:14:22.396 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:14:22.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:14:22.398 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:14:22.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:14:22.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:14:22.399 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:14:22.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:14:22.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:14:22.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:14:22.402 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:14:22.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:14:22.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:14:22.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:14:22.407 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:14:22.407 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:22.407 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:22.407 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:14:22.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:14:22.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:14:22.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2023-09-30T01:14:22.411 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:14:22.411 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:22.411 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:22.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53393 2023-09-30T01:14:22.414 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:14:22.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:14:22.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:14:22.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2023-09-30T01:14:22.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2023-09-30T01:14:22.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2023-09-30T01:14:22.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=c54ab296-c0fc-46b0-89d7-8bf95661b42e --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2023-09-30T01:14:22.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2023-09-30T01:14:22.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:14:22.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:14:22.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:14:22.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:14:22.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2023-09-30T01:14:22.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2023-09-30T01:14:22.771 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:14:22.771 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:14:22.772 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:14:22.773 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:14:22.773 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:22.773 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:22.773 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:14:22.773 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2023-09-30T01:14:23.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:14:23.079 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:14:23.079 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:14:23.079 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:14:23.079 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:14:23.082 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:14:23.082 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:14:23.082 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:14:23.082 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:14:23.082 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:14:23.083 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:23.083 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:23.083 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:14:23.083 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:14:23.083 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get fsid 2023-09-30T01:14:23.172 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:14:23.172 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:14:23.172 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:14:23.172 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:14:23.174 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:14:23.174 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:14:23.174 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:14:23.174 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:14:23.174 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:14:23.174 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:23.174 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:23.174 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.a.asok 2023-09-30T01:14:23.175 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:14:23.175 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.a.asok config get mon_host 2023-09-30T01:14:23.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2023-09-30T01:14:23.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:14:23.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:14:23.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2023-09-30T01:14:23.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:14:23.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2023-09-30T01:14:23.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2023-09-30T01:14:23.661 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:14:23.661 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:14:23.661 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:14:23.662 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:14:23.662 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:23.662 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:23.663 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:14:23.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2023-09-30T01:14:23.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:14:23.939 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2023-09-30T01:14:23.939 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:14:23.940 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2023-09-30T01:14:23.940 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:14:23.941 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:14:23.941 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2023-09-30T01:14:23.941 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2023-09-30T01:14:23.941 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2023-09-30T01:14:23.941 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:14:23.941 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:23.941 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:23.942 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.b.asok 2023-09-30T01:14:23.942 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:14:23.942 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.b.asok config get fsid 2023-09-30T01:14:24.032 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2023-09-30T01:14:24.032 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:14:24.032 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2023-09-30T01:14:24.032 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:14:24.032 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:14:24.033 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2023-09-30T01:14:24.033 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2023-09-30T01:14:24.033 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2023-09-30T01:14:24.034 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:14:24.034 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:24.034 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:24.034 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.b.asok 2023-09-30T01:14:24.034 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:14:24.034 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.b.asok config get mon_host 2023-09-30T01:14:24.120 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2023-09-30T01:14:24.120 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:14:24.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:14:24.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2023-09-30T01:14:24.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:14:24.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2023-09-30T01:14:24.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2023-09-30T01:14:24.528 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:14:24.529 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:14:24.529 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:14:24.529 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:14:24.529 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:24.530 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:24.530 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:14:24.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2023-09-30T01:14:24.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:14:24.821 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2023-09-30T01:14:24.821 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:14:24.821 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2023-09-30T01:14:24.821 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:14:24.821 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:14:24.822 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2023-09-30T01:14:24.822 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2023-09-30T01:14:24.822 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2023-09-30T01:14:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:14:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:24.823 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.c.asok 2023-09-30T01:14:24.824 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:14:24.824 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.c.asok config get fsid 2023-09-30T01:14:24.914 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2023-09-30T01:14:24.914 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:14:24.914 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2023-09-30T01:14:24.914 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:14:24.914 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:14:24.915 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2023-09-30T01:14:24.915 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2023-09-30T01:14:24.915 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2023-09-30T01:14:24.916 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:14:24.916 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:24.916 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:24.916 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.c.asok 2023-09-30T01:14:24.916 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:14:24.917 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.c.asok config get mon_host 2023-09-30T01:14:25.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2023-09-30T01:14:25.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:14:25.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:14:25.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2023-09-30T01:14:25.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:14:25.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2023-09-30T01:14:25.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2023-09-30T01:14:25.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:14:25.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:14:25.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:14:25.365 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:14:25.365 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:25.365 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:25.365 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:14:25.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2023-09-30T01:14:25.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:14:25.658 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2023-09-30T01:14:25.658 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:14:25.658 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2023-09-30T01:14:25.659 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:14:25.660 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:14:25.660 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2023-09-30T01:14:25.660 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2023-09-30T01:14:25.660 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2023-09-30T01:14:25.660 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:14:25.660 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:25.660 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:25.660 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.d.asok 2023-09-30T01:14:25.661 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:14:25.661 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.d.asok config get fsid 2023-09-30T01:14:25.765 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2023-09-30T01:14:25.765 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:14:25.765 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2023-09-30T01:14:25.765 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:14:25.765 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:14:25.766 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2023-09-30T01:14:25.766 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2023-09-30T01:14:25.766 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2023-09-30T01:14:25.767 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:14:25.767 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:25.767 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:25.767 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.d.asok 2023-09-30T01:14:25.767 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:14:25.767 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.d.asok config get mon_host 2023-09-30T01:14:25.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2023-09-30T01:14:25.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2023-09-30T01:14:25.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:14:25.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2023-09-30T01:14:25.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:14:25.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2023-09-30T01:14:25.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2023-09-30T01:14:26.271 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:14:26.271 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:14:26.271 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:14:26.272 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:14:26.272 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:26.272 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:26.272 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53393/$cluster-$name.asok' 2023-09-30T01:14:26.273 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.53393/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2023-09-30T01:14:26.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:14:26.545 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2023-09-30T01:14:26.545 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:14:26.546 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2023-09-30T01:14:26.546 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:14:26.546 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:14:26.547 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2023-09-30T01:14:26.547 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2023-09-30T01:14:26.547 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2023-09-30T01:14:26.548 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:14:26.548 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:26.548 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:26.548 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.e.asok 2023-09-30T01:14:26.549 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:14:26.549 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.e.asok config get fsid 2023-09-30T01:14:26.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2023-09-30T01:14:26.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:14:26.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2023-09-30T01:14:26.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:14:26.645 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:14:26.645 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2023-09-30T01:14:26.645 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2023-09-30T01:14:26.645 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2023-09-30T01:14:26.646 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:14:26.646 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:14:26.646 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:14:26.646 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53393/ceph-mon.e.asok 2023-09-30T01:14:26.647 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:14:26.647 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.53393/ceph-mon.e.asok config get mon_host 2023-09-30T01:14:26.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2023-09-30T01:14:26.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2023-09-30T01:14:26.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2023-09-30T01:14:26.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2023-09-30T01:14:26.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2023-09-30T01:14:26.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2023-09-30T01:14:26.732 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2023-09-30T01:14:26.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1696036526 2023-09-30T01:14:26.733 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:26.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036466 -lt 1696036526 ]] 2023-09-30T01:14:26.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2023-09-30T01:14:26.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:38.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:38.971 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:38.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2023-09-30T01:14:38.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:39.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2023-09-30T01:14:39.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2023-09-30T01:14:39.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2023-09-30T01:14:39.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2023-09-30T01:14:39.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2023-09-30T01:14:39.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2023-09-30T01:14:39.361 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,b,c,d,e) after stopping [dne] 2023-09-30T01:14:39.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2023-09-30T01:14:39.750 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2023-09-30T01:14:39.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2023-09-30T01:14:40.113 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2023-09-30T01:14:40.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2023-09-30T01:14:40.381 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2023-09-30T01:14:40.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2023-09-30T01:14:40.939 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2023-09-30T01:14:40.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2023-09-30T01:14:41.275 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2023-09-30T01:14:41.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2023-09-30T01:14:44.510 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2023-09-30T01:14:44.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2023-09-30T01:14:44.857 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2023-09-30T01:14:44.865 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2023-09-30T01:14:45.170 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2023-09-30T01:14:45.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2023-09-30T01:14:45.523 INFO:tasks.workunit.client.0.smithi186.stderr:adding another mon that is not yet online will not break quorum 2023-09-30T01:14:45.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2023-09-30T01:14:45.871 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.a 2023-09-30T01:14:45.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2023-09-30T01:14:46.203 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.b 2023-09-30T01:14:46.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2023-09-30T01:14:46.577 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.c 2023-09-30T01:14:46.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2023-09-30T01:14:46.872 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.d 2023-09-30T01:14:46.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2023-09-30T01:14:47.191 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.e 2023-09-30T01:14:47.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2023-09-30T01:14:47.202 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:14:47.202 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:14:47.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:14:47.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:14:47.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:14:47.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:14:47.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2023-09-30T01:14:47.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2023-09-30T01:14:47.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2023-09-30T01:14:47.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2023-09-30T01:14:47.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2023-09-30T01:14:47.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2023-09-30T01:14:47.315 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2023-09-30T01:14:47.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1696036547 2023-09-30T01:14:47.317 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:47.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036487 -lt 1696036547 ]] 2023-09-30T01:14:47.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:47.320 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:47.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:47.549 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:47.550 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:14:47.550 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:47.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:47.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:47.591 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:47.592 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036487 -lt 1696036547 ]] 2023-09-30T01:14:47.592 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:47.593 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:47.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:47.805 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:47.806 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:14:47.806 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:47.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:47.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:47.841 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:47.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036487 -lt 1696036547 ]] 2023-09-30T01:14:47.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:47.843 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:51.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:51.073 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:51.074 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:14:51.074 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:51.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:51.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:51.116 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:51.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036491 -lt 1696036547 ]] 2023-09-30T01:14:51.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:51.118 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:51.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:51.371 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:51.372 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:14:51.372 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:51.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:51.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:51.415 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:51.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036491 -lt 1696036547 ]] 2023-09-30T01:14:51.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:51.417 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:51.636 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:51.637 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:51.638 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:14:51.638 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:51.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:51.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:51.680 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:51.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036491 -lt 1696036547 ]] 2023-09-30T01:14:51.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:51.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:51.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:51.902 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:51.903 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:14:51.903 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:51.942 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:51.943 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:51.943 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:51.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036491 -lt 1696036547 ]] 2023-09-30T01:14:51.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:51.945 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:52.169 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:52.169 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:52.170 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:14:52.170 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:52.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:52.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:52.211 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:52.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036492 -lt 1696036547 ]] 2023-09-30T01:14:52.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:52.212 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:52.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:52.435 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:52.436 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:14:52.437 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:52.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:52.479 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:52.479 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:52.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036492 -lt 1696036547 ]] 2023-09-30T01:14:52.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:52.481 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:52.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:52.698 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:52.700 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:14:52.700 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:52.742 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:52.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:52.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:52.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036492 -lt 1696036547 ]] 2023-09-30T01:14:52.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:52.744 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:52.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:52.957 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:52.958 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:14:52.958 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:53.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:53.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:53.002 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:53.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036493 -lt 1696036547 ]] 2023-09-30T01:14:53.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:53.005 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:14:56.230 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:14:56.230 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:56.231 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:14:56.231 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:14:56.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:14:56.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:14:56.273 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:14:56.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036496 -lt 1696036547 ]] 2023-09-30T01:14:56.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2023-09-30T01:14:56.275 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:15:11.505 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:15:11.505 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:11.506 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:11.506 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2023-09-30T01:15:11.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2023-09-30T01:15:11.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2023-09-30T01:15:11.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2023-09-30T01:15:11.551 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2023-09-30T01:15:11.551 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2023-09-30T01:15:11.551 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2023-09-30T01:15:11.861 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2023-09-30T01:15:11.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2023-09-30T01:15:12.350 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (c,d,e) after stopping [b] 2023-09-30T01:15:12.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2023-09-30T01:15:15.580 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (b,d,e) after stopping [c] 2023-09-30T01:15:15.588 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2023-09-30T01:15:18.826 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (b,c,e) after stopping [d] 2023-09-30T01:15:18.835 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2023-09-30T01:15:19.215 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (b,c,d) after stopping [e] 2023-09-30T01:15:19.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2023-09-30T01:15:19.742 INFO:tasks.workunit.client.0.smithi186.stderr:adding another mon that is not yet online will not break quorum 2023-09-30T01:15:19.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2023-09-30T01:15:20.247 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.a 2023-09-30T01:15:20.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2023-09-30T01:15:20.567 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.b 2023-09-30T01:15:20.580 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2023-09-30T01:15:20.952 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.c 2023-09-30T01:15:20.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2023-09-30T01:15:21.366 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.d 2023-09-30T01:15:21.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2023-09-30T01:15:21.854 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.e 2023-09-30T01:15:21.863 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2023-09-30T01:15:21.864 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:15:21.864 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:15:21.864 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:15:21.864 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:15:21.865 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:15:21.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:15:21.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2023-09-30T01:15:21.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2023-09-30T01:15:21.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2023-09-30T01:15:21.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2023-09-30T01:15:21.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2023-09-30T01:15:21.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2023-09-30T01:15:21.978 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2023-09-30T01:15:21.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1696036581 2023-09-30T01:15:21.979 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:15:21.981 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036521 -lt 1696036581 ]] 2023-09-30T01:15:21.981 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:15:21.982 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:15:25.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:15:25.215 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:25.216 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:15:25.216 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:25.256 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:15:25.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:15:25.257 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:15:25.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036525 -lt 1696036581 ]] 2023-09-30T01:15:25.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:15:25.258 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:15:25.478 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:15:25.478 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:25.479 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:15:25.479 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:25.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:15:25.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:15:25.523 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:15:25.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036525 -lt 1696036581 ]] 2023-09-30T01:15:25.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:15:25.525 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:15:25.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:15:25.748 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:25.749 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:25.749 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:15:25.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:15:25.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:15:25.792 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:15:25.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036525 -lt 1696036581 ]] 2023-09-30T01:15:25.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:15:25.793 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:15:26.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:15:26.008 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:26.009 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:15:26.009 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:26.051 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:15:26.051 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:15:26.051 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:15:26.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036526 -lt 1696036581 ]] 2023-09-30T01:15:26.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:15:26.053 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:15:29.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:15:29.316 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:29.317 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:15:29.317 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:29.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:15:29.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:15:29.360 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:15:29.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036529 -lt 1696036581 ]] 2023-09-30T01:15:29.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:15:29.362 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:15:29.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:15:29.590 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:29.591 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:15:29.591 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:29.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:15:29.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:15:29.633 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:15:29.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036529 -lt 1696036581 ]] 2023-09-30T01:15:29.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:15:29.635 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:15:29.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:15:29.859 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:29.860 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:15:29.860 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:29.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2023-09-30T01:15:29.904 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2023-09-30T01:15:29.904 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2023-09-30T01:15:29.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1696036529 -lt 1696036581 ]] 2023-09-30T01:15:29.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2023-09-30T01:15:29.905 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2023-09-30T01:15:39.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2023-09-30T01:15:39.133 INFO:tasks.workunit.client.0.smithi186.stderr:{"election_epoch":18,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:39.133 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":18,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c54ab296-c0fc-46b0-89d7-8bf95661b42e","modified":"2023-09-30T01:14:22.439280Z","created":"2023-09-30T01:14:22.439280Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2023-09-30T01:15:39.134 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2023-09-30T01:15:39.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2023-09-30T01:15:39.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2023-09-30T01:15:39.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2023-09-30T01:15:39.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2023-09-30T01:15:39.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2023-09-30T01:15:39.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2023-09-30T01:15:42.404 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (b,c,d) after stopping [a] 2023-09-30T01:15:42.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2023-09-30T01:15:42.699 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2023-09-30T01:15:42.701 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2023-09-30T01:15:45.873 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2023-09-30T01:15:45.876 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2023-09-30T01:15:52.057 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2023-09-30T01:15:52.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2023-09-30T01:15:52.360 INFO:tasks.workunit.client.0.smithi186.stderr:quorum should be preserved (b,c,d) after stopping [e] 2023-09-30T01:15:52.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2023-09-30T01:15:52.616 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2023-09-30T01:15:52.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2023-09-30T01:15:52.927 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.a 2023-09-30T01:15:52.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2023-09-30T01:15:53.187 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: removing mon.b would break quorum 2023-09-30T01:15:53.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2023-09-30T01:15:56.365 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: removing mon.c would break quorum 2023-09-30T01:15:56.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2023-09-30T01:15:59.536 INFO:tasks.workunit.client.0.smithi186.stderr:Error EBUSY: removing mon.d would break quorum 2023-09-30T01:15:59.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2023-09-30T01:15:59.842 INFO:tasks.workunit.client.0.smithi186.stderr:safe to remove mon.e 2023-09-30T01:15:59.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2023-09-30T01:15:59.851 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:15:59.851 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:15:59.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:15:59.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:15:59.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:15:59.966 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:15:59.966 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2023-09-30T01:15:59.966 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:15:59.966 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:15:59.966 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:15:59.966 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:15:59.967 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:15:59.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:15:59.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:15:59.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:15:59.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:15:59.981 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:15:59.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:15:59.983 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:15:59.984 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:15:59.984 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:15:59.985 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:15:59.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:15:59.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:15:59.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:15:59.986 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:15:59.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:15:59.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:15:59.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:16:00.003 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:16:00.004 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:16:00.004 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:16:00.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:16:00.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:16:00.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:16:00.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: main: code=0 2023-09-30T01:16:00.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: main: teardown td/ok-to-stop 0 2023-09-30T01:16:00.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2023-09-30T01:16:00.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-09-30T01:16:00.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2023-09-30T01:16:00.007 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:16:00.007 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:16:00.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:16:00.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:16:00.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:16:00.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:16:00.011 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:16:00.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:16:00.012 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:16:00.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:16:00.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:16:00.015 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:16:00.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:16:00.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:16:00.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:16:00.017 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:16:00.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:16:00.020 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-09-30T01:16:00.020 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2023-09-30T01:16:00.022 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:16:00.022 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:16:00.022 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53393 2023-09-30T01:16:00.022 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53393 2023-09-30T01:16:00.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:16:00.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:16:00.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: main: return 0 2023-09-30T01:16:00.026 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-30T01:16:00.026 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-30T01:16:00.043 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2023-09-30T01:16:00.043 DEBUG:teuthology.orchestra.run.smithi186:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=93dce5e17918f8aca65993a7a2dc40233a47b481 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2023-09-30T01:16:00.103 INFO:tasks.workunit.client.0.smithi186.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-09-30T01:16:00.116 INFO:tasks.workunit.client.0.smithi186.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-09-30T01:16:00.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-09-30T01:16:00.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-09-30T01:16:00.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: export PYTHONWARNINGS=ignore 2023-09-30T01:16:00.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: main: PYTHONWARNINGS=ignore 2023-09-30T01:16:00.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: export CEPH_CONF=/dev/null 2023-09-30T01:16:00.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: main: CEPH_CONF=/dev/null 2023-09-30T01:16:00.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: main: unset CEPH_ARGS 2023-09-30T01:16:00.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: main: local code 2023-09-30T01:16:00.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: main: run td/rados-striper 2023-09-30T01:16:00.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2023-09-30T01:16:00.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2023-09-30T01:16:00.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2023-09-30T01:16:00.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2023-09-30T01:16:00.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2023-09-30T01:16:00.118 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2023-09-30T01:16:00.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=dbc0551b-9373-4ddb-8a50-6fb2ac1fe276 --auth-supported=none ' 2023-09-30T01:16:00.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2023-09-30T01:16:00.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2023-09-30T01:16:00.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2023-09-30T01:16:00.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2023-09-30T01:16:00.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2023-09-30T01:16:00.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:16:00.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2023-09-30T01:16:00.119 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:16:00.119 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:16:00.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:16:00.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:16:00.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:16:00.122 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:16:00.123 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:16:00.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:16:00.125 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:16:00.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:16:00.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:16:00.127 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:16:00.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:16:00.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:16:00.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:16:00.129 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:16:00.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:16:00.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:16:00.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2023-09-30T01:16:00.135 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:16:00.135 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:16:00.135 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70623 2023-09-30T01:16:00.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70623 2023-09-30T01:16:00.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:16:00.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:16:00.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2023-09-30T01:16:00.139 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:16:00.139 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:16:00.139 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70623 2023-09-30T01:16:00.139 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70623 2023-09-30T01:16:00.142 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:16:00.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-09-30T01:16:00.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-09-30T01:16:00.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:16:00.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2023-09-30T01:16:00.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2023-09-30T01:16:00.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2023-09-30T01:16:00.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:16:00.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:16:00.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:16:00.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2023-09-30T01:16:00.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2023-09-30T01:16:00.544 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:16:00.544 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:16:00.544 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:16:00.545 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:16:00.545 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:16:00.545 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70623 2023-09-30T01:16:00.545 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70623/$cluster-$name.asok' 2023-09-30T01:16:00.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.70623/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:16:00.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:16:00.824 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:16:00.824 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:16:00.824 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:16:00.824 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:16:00.825 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:16:00.825 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:16:00.825 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:16:00.826 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:16:00.826 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:16:00.826 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:16:00.826 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70623 2023-09-30T01:16:00.827 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70623/ceph-mon.a.asok 2023-09-30T01:16:00.827 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:16:00.827 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70623/ceph-mon.a.asok config get fsid 2023-09-30T01:16:00.920 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:16:00.920 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:16:00.921 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:16:00.921 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:16:00.922 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:16:00.922 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:16:00.922 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:16:00.922 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:16:00.922 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:16:00.922 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:16:00.922 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70623 2023-09-30T01:16:00.922 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70623/ceph-mon.a.asok 2023-09-30T01:16:00.923 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:16:00.923 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70623/ceph-mon.a.asok config get mon_host 2023-09-30T01:16:01.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2023-09-30T01:16:01.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2023-09-30T01:16:01.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:16:01.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:16:01.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:16:01.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2023-09-30T01:16:01.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dbc0551b-9373-4ddb-8a50-6fb2ac1fe276 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2023-09-30T01:16:01.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:16:01.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:16:01.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:16:01.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2023-09-30T01:16:01.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2023-09-30T01:16:01.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:16:01.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:16:01.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2023-09-30T01:16:01.006 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:16:01.006 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:16:01.006 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:16:01.007 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:16:01.007 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:16:01.007 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70623 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70623/$cluster-$name.asok' 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70623/$cluster-$name.asok' 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:16:01.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:16:01.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:16:01.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2023-09-30T01:16:01.009 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:16:01.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ed7b2478-fd12-4854-9bea-80aab8a26679 2023-09-30T01:16:01.009 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 ed7b2478-fd12-4854-9bea-80aab8a26679 2023-09-30T01:16:01.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ed7b2478-fd12-4854-9bea-80aab8a26679' 2023-09-30T01:16:01.010 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:16:01.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRdhdl6WllARAAGVQyuCxKPh6eyWawzsEYzA== 2023-09-30T01:16:01.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRdhdl6WllARAAGVQyuCxKPh6eyWawzsEYzA=="}' 2023-09-30T01:16:01.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ed7b2478-fd12-4854-9bea-80aab8a26679 -i td/rados-striper/0/new.json 2023-09-30T01:16:01.992 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:16:02.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2023-09-30T01:16:02.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dbc0551b-9373-4ddb-8a50-6fb2ac1fe276 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70623/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRdhdl6WllARAAGVQyuCxKPh6eyWawzsEYzA== --osd-uuid ed7b2478-fd12-4854-9bea-80aab8a26679 2023-09-30T01:16:02.037 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:02.033+0000 7facb46023c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:02.040 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:02.033+0000 7facb46023c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:02.042 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:02.037+0000 7facb46023c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:02.043 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:02.037+0000 7facb46023c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2023-09-30T01:16:02.043 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:02.037+0000 7facb46023c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2023-09-30T01:16:02.043 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:02.037+0000 7facb46023c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2023-09-30T01:16:02.131 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:02.125+0000 7facb46023c0 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2023-09-30T01:16:07.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2023-09-30T01:16:07.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:16:07.889 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:16:07.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:16:07.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:16:08.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:16:08.129 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:16:08.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:16:08.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dbc0551b-9373-4ddb-8a50-6fb2ac1fe276 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70623/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:16:08.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:16:08.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:16:08.158 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:08.153+0000 7fc60a7893c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:08.160 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:08.153+0000 7fc60a7893c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:08.162 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:08.157+0000 7fc60a7893c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:08.355 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:16:08.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:16:08.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:16:08.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:16:08.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:16:08.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:08.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:16:08.356 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:16:08.357 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:08.357 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:08.572 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:08.708 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:08.701+0000 7fc60a7893c0 -1 Falling back to public interface 2023-09-30T01:16:09.574 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:09.574 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:09.574 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:16:09.574 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:16:09.575 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:09.575 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:09.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:10.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:10.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:10.797 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:16:10.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:16:10.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:10.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:11.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:11.401 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:11.397+0000 7fc60a7893c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:16:12.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:12.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:12.018 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:16:12.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:16:12.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:12.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:12.406 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:13.407 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:13.408 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:13.408 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:16:13.408 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:16:13.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:13.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:13.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:14.633 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:14.633 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:14.633 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:16:14.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:16:14.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:14.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:14.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:15.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:15.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:15.861 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:16:15.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:16:15.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:15.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:16.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:17.086 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:17.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:17.087 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:16:17.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:16:17.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:17.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:17.313 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:18.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:18.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:18.315 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:16:18.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:16:18.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:18.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:18.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:19.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:19.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:19.537 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:16:19.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:16:19.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:19.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:19.762 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:20.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:20.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:20.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:16:20.765 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:16:20.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:20.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:21.023 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:22.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:22.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:22.025 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:16:22.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:16:22.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:22.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:22.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:23.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:23.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:23.253 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:16:23.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:16:23.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:23.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:23.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:24.469 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:24.469 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:24.470 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:16:24.470 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:16:24.470 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:24.471 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:24.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:25.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:25.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:25.691 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:16:25.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:16:25.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:25.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:25.922 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:26.924 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:26.924 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:26.924 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:16:26.925 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:16:26.925 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:26.925 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:27.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:28.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:28.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:28.153 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:16:28.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:16:28.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:28.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:28.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:29.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:29.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:29.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-09-30T01:16:29.374 INFO:tasks.workunit.client.0.smithi186.stdout:17 2023-09-30T01:16:29.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:29.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:29.595 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:30.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:30.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:30.597 INFO:tasks.workunit.client.0.smithi186.stdout:18 2023-09-30T01:16:30.597 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-09-30T01:16:30.597 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:30.597 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:30.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:31.820 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:31.821 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:31.821 INFO:tasks.workunit.client.0.smithi186.stdout:19 2023-09-30T01:16:31.821 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2023-09-30T01:16:31.821 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:31.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:32.038 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:33.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:33.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:33.041 INFO:tasks.workunit.client.0.smithi186.stdout:20 2023-09-30T01:16:33.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2023-09-30T01:16:33.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:33.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:16:33.263 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/71569,v1:127.0.0.1:6801/71569] [v2:127.0.0.1:6802/71569,v1:127.0.0.1:6803/71569] exists,up ed7b2478-fd12-4854-9bea-80aab8a26679 2023-09-30T01:16:33.264 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:16:33.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:16:33.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:16:33.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2023-09-30T01:16:33.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2023-09-30T01:16:33.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:16:33.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-09-30T01:16:33.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:16:33.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2023-09-30T01:16:33.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dbc0551b-9373-4ddb-8a50-6fb2ac1fe276 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2023-09-30T01:16:33.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:16:33.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:16:33.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:16:33.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2023-09-30T01:16:33.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2023-09-30T01:16:33.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:16:33.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:16:33.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2023-09-30T01:16:33.266 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:16:33.266 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:16:33.266 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:16:33.268 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:16:33.268 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:16:33.268 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70623 2023-09-30T01:16:33.268 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70623/$cluster-$name.asok' 2023-09-30T01:16:33.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70623/$cluster-$name.asok' 2023-09-30T01:16:33.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:16:33.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:16:33.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:16:33.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2023-09-30T01:16:33.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2023-09-30T01:16:33.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:16:33.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:16:33.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:16:33.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:16:33.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:16:33.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:16:33.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2023-09-30T01:16:33.269 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:16:33.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2af34b7b-bab7-423d-bf99-ba72308e6626 2023-09-30T01:16:33.270 INFO:tasks.workunit.client.0.smithi186.stdout:add osd1 2af34b7b-bab7-423d-bf99-ba72308e6626 2023-09-30T01:16:33.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2af34b7b-bab7-423d-bf99-ba72308e6626' 2023-09-30T01:16:33.270 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:16:33.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDxdhdlSmAcERAAABnkqvkaLpSFP6z3b0ZaNQ== 2023-09-30T01:16:33.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDxdhdlSmAcERAAABnkqvkaLpSFP6z3b0ZaNQ=="}' 2023-09-30T01:16:33.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2af34b7b-bab7-423d-bf99-ba72308e6626 -i td/rados-striper/1/new.json 2023-09-30T01:16:33.716 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:16:33.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2023-09-30T01:16:33.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=dbc0551b-9373-4ddb-8a50-6fb2ac1fe276 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70623/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxdhdlSmAcERAAABnkqvkaLpSFP6z3b0ZaNQ== --osd-uuid 2af34b7b-bab7-423d-bf99-ba72308e6626 2023-09-30T01:16:33.761 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:33.752+0000 7ff83c9ec3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:33.808 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:33.800+0000 7ff83c9ec3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:33.810 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:33.804+0000 7ff83c9ec3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:33.811 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:33.804+0000 7ff83c9ec3c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2023-09-30T01:16:33.811 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:33.804+0000 7ff83c9ec3c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2023-09-30T01:16:33.812 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:33.804+0000 7ff83c9ec3c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2023-09-30T01:16:33.855 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:33.848+0000 7ff83c9ec3c0 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2023-09-30T01:16:39.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2023-09-30T01:16:39.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:16:39.789 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd1 key to auth repository 2023-09-30T01:16:39.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-09-30T01:16:39.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:16:40.021 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:16:40.022 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-09-30T01:16:40.022 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=dbc0551b-9373-4ddb-8a50-6fb2ac1fe276 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70623/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:16:40.022 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:16:40.023 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:16:40.023 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:16:40.044 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:40.036+0000 7faa33f2f3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:40.049 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:40.044+0000 7faa33f2f3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:40.051 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:40.044+0000 7faa33f2f3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:16:40.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-09-30T01:16:40.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:16:40.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:16:40.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:16:40.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:16:40.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:40.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:16:40.244 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:16:40.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:40.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:40.461 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:40.600 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:40.592+0000 7faa33f2f3c0 -1 Falling back to public interface 2023-09-30T01:16:41.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:41.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:41.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:16:41.463 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:16:41.464 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:41.464 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:41.687 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:42.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:42.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:42.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:16:42.689 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:16:42.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:42.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:42.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:43.256 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:16:43.248+0000 7faa33f2f3c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:16:43.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:43.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:43.909 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:16:43.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:16:43.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:43.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:44.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:45.271 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:45.271 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:45.271 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:16:45.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:16:45.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:45.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:45.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:46.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:46.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:46.497 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:16:46.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:16:46.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:46.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:46.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:47.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:47.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:47.716 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:16:47.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:16:47.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:47.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:47.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:48.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:48.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:48.946 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:16:48.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:16:48.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:48.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:49.169 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:50.171 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:50.171 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:50.171 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:16:50.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:16:50.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:50.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:50.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:51.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:51.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:51.395 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:16:51.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:16:51.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:51.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:51.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:52.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:52.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:52.690 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:16:52.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:16:52.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:52.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:52.915 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:53.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:53.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:53.918 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:16:53.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:16:53.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:53.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:54.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:55.139 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:55.139 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:55.139 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:16:55.140 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:16:55.140 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:55.140 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:55.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:56.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:56.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:56.362 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:16:56.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:16:56.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:56.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:56.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:57.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:57.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:57.693 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:16:57.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:16:57.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:57.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:58.434 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:16:59.436 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:16:59.436 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:16:59.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:16:59.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:16:59.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:16:59.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:16:59.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:00.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:00.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:00.663 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:17:00.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:17:00.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:00.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:17:00.896 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:01.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:01.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:01.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-09-30T01:17:01.898 INFO:tasks.workunit.client.0.smithi186.stdout:17 2023-09-30T01:17:01.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:01.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:17:02.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:03.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:03.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:03.119 INFO:tasks.workunit.client.0.smithi186.stdout:18 2023-09-30T01:17:03.120 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-09-30T01:17:03.120 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:03.120 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:17:03.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:04.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:04.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:04.341 INFO:tasks.workunit.client.0.smithi186.stdout:19 2023-09-30T01:17:04.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2023-09-30T01:17:04.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:04.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:17:04.636 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/73422,v1:127.0.0.1:6809/73422] [v2:127.0.0.1:6810/73422,v1:127.0.0.1:6811/73422] exists,up 2af34b7b-bab7-423d-bf99-ba72308e6626 2023-09-30T01:17:04.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:17:04.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:17:04.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:17:04.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2023-09-30T01:17:04.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2023-09-30T01:17:04.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dbc0551b-9373-4ddb-8a50-6fb2ac1fe276 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:17:04.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2023-09-30T01:17:04.641 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:17:04.641 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:17:04.641 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:17:04.641 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:17:04.642 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:17:04.642 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70623 2023-09-30T01:17:04.642 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70623/$cluster-$name.asok' 2023-09-30T01:17:04.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70623/$cluster-$name.asok' 2023-09-30T01:17:04.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:17:04.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:17:04.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:17:04.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2023-09-30T01:17:04.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2023-09-30T01:17:04.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:17:04.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:17:04.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:17:04.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:17:04.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:17:04.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:17:04.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2023-09-30T01:17:04.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:17:04.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5d504396-b9e6-4c63-b5e9-8805daf33a3f 2023-09-30T01:17:04.645 INFO:tasks.workunit.client.0.smithi186.stdout:add osd2 5d504396-b9e6-4c63-b5e9-8805daf33a3f 2023-09-30T01:17:04.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5d504396-b9e6-4c63-b5e9-8805daf33a3f' 2023-09-30T01:17:04.646 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:17:04.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQdxdltx2uJxAA6zOgSgXNEoSxuJdVUvPMUQ== 2023-09-30T01:17:04.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQdxdltx2uJxAA6zOgSgXNEoSxuJdVUvPMUQ=="}' 2023-09-30T01:17:04.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5d504396-b9e6-4c63-b5e9-8805daf33a3f -i td/rados-striper/2/new.json 2023-09-30T01:17:04.994 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:17:05.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2023-09-30T01:17:05.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=dbc0551b-9373-4ddb-8a50-6fb2ac1fe276 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70623/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQdxdltx2uJxAA6zOgSgXNEoSxuJdVUvPMUQ== --osd-uuid 5d504396-b9e6-4c63-b5e9-8805daf33a3f 2023-09-30T01:17:05.043 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:05.036+0000 7fd3aed9b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:05.079 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:05.072+0000 7fd3aed9b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:05.081 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:05.072+0000 7fd3aed9b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:05.081 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:05.076+0000 7fd3aed9b3c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2023-09-30T01:17:05.081 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:05.076+0000 7fd3aed9b3c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2023-09-30T01:17:05.082 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:05.076+0000 7fd3aed9b3c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2023-09-30T01:17:05.126 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:05.120+0000 7fd3aed9b3c0 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2023-09-30T01:17:11.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2023-09-30T01:17:11.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:17:11.085 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd2 key to auth repository 2023-09-30T01:17:11.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-09-30T01:17:11.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:17:11.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-09-30T01:17:11.315 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.2 2023-09-30T01:17:11.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:17:11.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=dbc0551b-9373-4ddb-8a50-6fb2ac1fe276 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.70623/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:17:11.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:17:11.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:17:11.345 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:11.336+0000 7f577e69e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:11.348 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:11.340+0000 7f577e69e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:11.349 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:11.344+0000 7f577e69e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:11.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-09-30T01:17:11.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:17:11.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-09-30T01:17:11.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:17:11.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:17:11.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:11.529 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:17:11.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:17:11.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:11.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:11.784 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:11.908 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:11.900+0000 7f577e69e3c0 -1 Falling back to public interface 2023-09-30T01:17:12.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:12.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:12.786 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:17:12.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:17:12.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:12.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:13.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:14.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:14.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:14.009 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:17:14.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:17:14.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:14.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:14.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:14.547 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:14.540+0000 7f577e69e3c0 -1 osd.2 0 log_to_monitors true 2023-09-30T01:17:15.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:15.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:15.245 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:17:15.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:17:15.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:15.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:15.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:16.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:16.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:16.645 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:17:16.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:17:16.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:16.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:16.866 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:17.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:17.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:17.869 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:17:17.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:17:17.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:17.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:18.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:19.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:19.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:19.095 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:17:19.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:17:19.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:19.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:19.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:20.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:20.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:20.314 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:17:20.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:17:20.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:20.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:20.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:21.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:21.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:21.536 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:17:21.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:17:21.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:21.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:21.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:22.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:22.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:22.766 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:17:22.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:17:22.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:22.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:22.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:23.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:23.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:23.995 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:17:23.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:17:23.996 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:23.996 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:24.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:25.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:25.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:25.226 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:17:25.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:17:25.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:25.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:25.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:26.459 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:26.459 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:26.459 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:17:26.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:17:26.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:26.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:26.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:27.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:27.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:27.690 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:17:27.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:17:27.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:27.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:27.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:28.921 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:28.921 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:28.921 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:17:28.921 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:17:28.922 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:28.922 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:29.142 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:30.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:30.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:30.144 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:17:30.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:17:30.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:30.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:30.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:31.365 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:31.365 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:31.365 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:17:31.365 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:17:31.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:31.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:31.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:32.591 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:32.592 INFO:tasks.workunit.client.0.smithi186.stdout:17 2023-09-30T01:17:32.592 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:32.592 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-09-30T01:17:32.592 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:32.593 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:32.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:33.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:33.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:33.818 INFO:tasks.workunit.client.0.smithi186.stdout:18 2023-09-30T01:17:33.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-09-30T01:17:33.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:33.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:34.043 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:35.044 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:35.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:35.045 INFO:tasks.workunit.client.0.smithi186.stdout:19 2023-09-30T01:17:35.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2023-09-30T01:17:35.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:35.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:35.264 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:36.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:36.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:36.266 INFO:tasks.workunit.client.0.smithi186.stdout:20 2023-09-30T01:17:36.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2023-09-30T01:17:36.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:36.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:17:36.515 INFO:tasks.workunit.client.0.smithi186.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/75254,v1:127.0.0.1:6817/75254] [v2:127.0.0.1:6818/75254,v1:127.0.0.1:6819/75254] exists,up 5d504396-b9e6-4c63-b5e9-8805daf33a3f 2023-09-30T01:17:36.516 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:17:36.516 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:17:36.516 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:17:36.516 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2023-09-30T01:17:36.516 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:17:36.779 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:17:36.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:17:36.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:17:37.158 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:17:37.169 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:17:38.171 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:17:40.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2023-09-30T01:17:40.197 INFO:tasks.workunit.client.0.smithi186.stderr:1+0 records in 2023-09-30T01:17:40.198 INFO:tasks.workunit.client.0.smithi186.stderr:1+0 records out 2023-09-30T01:17:40.198 INFO:tasks.workunit.client.0.smithi186.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 0.000212106 s, 5.8 MB/s 2023-09-30T01:17:40.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2023-09-30T01:17:40.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2023-09-30T01:17:40.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2023-09-30T01:17:40.945 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2023-09-30T01:17:40.945 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2023-09-30T01:17:40.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2023-09-30T01:17:40.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2023-09-30T01:17:40.996 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2023-09-30T01:17:40.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2023-09-30T01:17:41.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2023-09-30T01:17:41.035 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2023-09-30T01:17:41.273 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2023-09-30T01:17:41.275 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2023-09-30T01:17:41.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2023-09-30T01:17:41.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2023-09-30T01:17:41.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2023-09-30T01:17:41.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2023-09-30T01:17:41.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2023-09-30T01:17:41.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2023-09-30T01:17:41.796 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2023-09-30T01:17:41.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2023-09-30T01:17:41.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2023-09-30T01:17:41.799 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2023-09-30T01:17:41.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2023-09-30T01:17:41.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2023-09-30T01:17:42.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2023-09-30T01:17:42.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2023-09-30T01:17:42.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2023-09-30T01:17:42.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2023-09-30T01:17:42.050 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2023-09-30T01:17:42.050 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2023-09-30T01:17:42.052 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2023-09-30T01:17:42.052 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2023-09-30T01:17:42.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2023-09-30T01:17:42.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2023-09-30T01:17:42.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2023-09-30T01:17:42.236 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2023-09-30T01:17:42.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2023-09-30T01:17:42.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2023-09-30T01:17:42.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/rados-striper 2023-09-30T01:17:42.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2023-09-30T01:17:42.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local 'expected=No data available' 2023-09-30T01:17:42.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2023-09-30T01:17:42.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2023-09-30T01:17:42.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2023-09-30T01:17:42.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2023-09-30T01:17:42.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2023-09-30T01:17:42.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2023-09-30T01:17:42.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: expect_failure: return 0 2023-09-30T01:17:42.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2023-09-30T01:17:42.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/rados-striper 2023-09-30T01:17:42.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2023-09-30T01:17:42.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local 'expected=No data available' 2023-09-30T01:17:42.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2023-09-30T01:17:42.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2023-09-30T01:17:42.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2023-09-30T01:17:42.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2023-09-30T01:17:42.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2023-09-30T01:17:42.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2023-09-30T01:17:42.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: expect_failure: return 0 2023-09-30T01:17:42.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2023-09-30T01:17:42.578 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2023-09-30T01:17:42.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/rados-striper 2023-09-30T01:17:42.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2023-09-30T01:17:42.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local 'expected=No such file or directory' 2023-09-30T01:17:42.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2023-09-30T01:17:42.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2023-09-30T01:17:42.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: rados --pool rbd --striper stat toyfile 2023-09-30T01:17:42.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2023-09-30T01:17:42.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2023-09-30T01:17:42.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2023-09-30T01:17:42.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: expect_failure: return 0 2023-09-30T01:17:42.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2023-09-30T01:17:42.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/rados-striper 2023-09-30T01:17:42.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2023-09-30T01:17:42.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local 'expected=No such file or directory' 2023-09-30T01:17:42.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2023-09-30T01:17:42.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2023-09-30T01:17:42.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2023-09-30T01:17:42.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2023-09-30T01:17:42.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2023-09-30T01:17:42.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2023-09-30T01:17:42.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: expect_failure: return 0 2023-09-30T01:17:42.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2023-09-30T01:17:42.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2023-09-30T01:17:42.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:17:42.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2023-09-30T01:17:42.656 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:17:42.656 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:17:42.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:17:42.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:17:42.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:17:42.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:17:42.771 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:17:42.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:17:42.773 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:17:42.775 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:17:42.775 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:17:42.776 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:17:42.777 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:17:42.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:17:42.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:17:42.779 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:17:42.780 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:17:42.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:17:42.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2023-09-30T01:17:42.800 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:17:42.800 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:17:42.800 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70623 2023-09-30T01:17:42.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70623 2023-09-30T01:17:42.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:17:42.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:17:42.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: main: code=0 2023-09-30T01:17:42.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: main: teardown td/rados-striper 0 2023-09-30T01:17:42.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2023-09-30T01:17:42.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-09-30T01:17:42.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2023-09-30T01:17:42.803 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:17:42.804 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:17:42.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:17:42.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:17:42.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:17:42.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:17:42.808 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:17:42.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:17:42.811 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:17:42.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:17:42.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:17:42.813 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:17:42.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:17:42.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:17:42.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:17:42.815 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:17:42.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:17:42.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-09-30T01:17:42.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2023-09-30T01:17:42.821 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:17:42.821 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:17:42.822 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70623 2023-09-30T01:17:42.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70623 2023-09-30T01:17:42.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:17:42.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:17:42.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: main: return 0 2023-09-30T01:17:42.824 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-30T01:17:42.824 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-30T01:17:42.838 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2023-09-30T01:17:42.839 DEBUG:teuthology.orchestra.run.smithi186:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=93dce5e17918f8aca65993a7a2dc40233a47b481 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2023-09-30T01:17:42.858 INFO:tasks.workunit.client.0.smithi186.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-09-30T01:17:42.870 INFO:tasks.workunit.client.0.smithi186.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-09-30T01:17:42.870 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: run_tests: export .:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2023-09-30T01:17:42.870 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2260: export: `.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin': not a valid identifier 2023-09-30T01:17:42.870 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: run_tests: export CEPH_MON=127.0.0.1:7109 2023-09-30T01:17:42.870 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: run_tests: CEPH_MON=127.0.0.1:7109 2023-09-30T01:17:42.870 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: run_tests: export CEPH_ARGS 2023-09-30T01:17:42.871 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: run_tests: uuidgen 2023-09-30T01:17:42.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: run_tests: CEPH_ARGS+=' --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none ' 2023-09-30T01:17:42.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2023-09-30T01:17:42.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: run_tests: export CEPH_CONF=/dev/null 2023-09-30T01:17:42.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: run_tests: CEPH_CONF=/dev/null 2023-09-30T01:17:42.873 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: run_tests: set 2023-09-30T01:17:42.873 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2023-09-30T01:17:42.877 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: run_tests: local 'funcs=test_activate_osd 2023-09-30T01:17:42.877 INFO:tasks.workunit.client.0.smithi186.stderr:test_activate_osd_after_mark_down 2023-09-30T01:17:42.877 INFO:tasks.workunit.client.0.smithi186.stderr:test_activate_osd_skip_benchmark 2023-09-30T01:17:42.877 INFO:tasks.workunit.client.0.smithi186.stderr:test_destroy_osd 2023-09-30T01:17:42.878 INFO:tasks.workunit.client.0.smithi186.stderr:test_display_logs 2023-09-30T01:17:42.878 INFO:tasks.workunit.client.0.smithi186.stderr:test_erasure_code_plugin_exists 2023-09-30T01:17:42.878 INFO:tasks.workunit.client.0.smithi186.stderr:test_expect_failure 2023-09-30T01:17:42.878 INFO:tasks.workunit.client.0.smithi186.stderr:test_flush_pg_stats 2023-09-30T01:17:42.878 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_config 2023-09-30T01:17:42.878 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_is_making_recovery_progress 2023-09-30T01:17:42.878 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_last_scrub_stamp 2023-09-30T01:17:42.878 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_not_primary 2023-09-30T01:17:42.878 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_num_active_clean 2023-09-30T01:17:42.879 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_num_active_or_peered 2023-09-30T01:17:42.879 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_num_pgs 2023-09-30T01:17:42.879 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_op_scheduler 2023-09-30T01:17:42.879 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_osd_id_used_by_pgs 2023-09-30T01:17:42.879 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_osds 2023-09-30T01:17:42.879 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_pg 2023-09-30T01:17:42.879 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_primary 2023-09-30T01:17:42.879 INFO:tasks.workunit.client.0.smithi186.stderr:test_get_timeout_delays 2023-09-30T01:17:42.879 INFO:tasks.workunit.client.0.smithi186.stderr:test_is_clean 2023-09-30T01:17:42.879 INFO:tasks.workunit.client.0.smithi186.stderr:test_kill_daemon 2023-09-30T01:17:42.880 INFO:tasks.workunit.client.0.smithi186.stderr:test_kill_daemons 2023-09-30T01:17:42.880 INFO:tasks.workunit.client.0.smithi186.stderr:test_objectstore_tool 2023-09-30T01:17:42.880 INFO:tasks.workunit.client.0.smithi186.stderr:test_pg_scrub 2023-09-30T01:17:42.880 INFO:tasks.workunit.client.0.smithi186.stderr:test_repair 2023-09-30T01:17:42.880 INFO:tasks.workunit.client.0.smithi186.stderr:test_run_in_background 2023-09-30T01:17:42.880 INFO:tasks.workunit.client.0.smithi186.stderr:test_run_mon 2023-09-30T01:17:42.880 INFO:tasks.workunit.client.0.smithi186.stderr:test_run_osd 2023-09-30T01:17:42.880 INFO:tasks.workunit.client.0.smithi186.stderr:test_set_config 2023-09-30T01:17:42.880 INFO:tasks.workunit.client.0.smithi186.stderr:test_setup 2023-09-30T01:17:42.880 INFO:tasks.workunit.client.0.smithi186.stderr:test_teardown 2023-09-30T01:17:42.881 INFO:tasks.workunit.client.0.smithi186.stderr:test_wait_background 2023-09-30T01:17:42.881 INFO:tasks.workunit.client.0.smithi186.stderr:test_wait_for_clean 2023-09-30T01:17:42.881 INFO:tasks.workunit.client.0.smithi186.stderr:test_wait_for_health_ok 2023-09-30T01:17:42.881 INFO:tasks.workunit.client.0.smithi186.stderr:test_wait_for_osd 2023-09-30T01:17:42.881 INFO:tasks.workunit.client.0.smithi186.stderr:test_wait_for_peered 2023-09-30T01:17:42.881 INFO:tasks.workunit.client.0.smithi186.stderr:test_wait_for_scrub 2023-09-30T01:17:42.881 INFO:tasks.workunit.client.0.smithi186.stderr:test_wait_osd_id_used_by_pgs' 2023-09-30T01:17:42.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: run_tests: local dir=td/ceph-helpers 2023-09-30T01:17:42.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:17:42.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_activate_osd td/ceph-helpers 2023-09-30T01:17:42.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2023-09-30T01:17:42.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2023-09-30T01:17:42.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:17:42.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:17:42.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:17:42.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:17:42.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:17:42.883 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:17:42.883 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:17:42.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:17:42.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:17:42.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:17:42.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:17:42.884 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:17:42.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:17:42.884 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:17:42.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:17:42.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:17:42.885 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:17:42.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:17:42.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:17:42.888 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:17:42.888 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:17:42.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:17:42.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:17:42.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:17:42.893 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:17:42.893 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:17:42.893 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:17:42.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:17:42.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:17:42.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:17:42.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:17:42.897 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:17:42.897 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:17:42.897 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:17:42.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:17:42.899 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:17:42.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-09-30T01:17:42.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-09-30T01:17:42.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:17:42.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:17:42.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2023-09-30T01:17:42.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:17:42.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:17:42.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:17:42.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:17:42.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:17:42.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2023-09-30T01:17:43.295 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:17:43.295 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:17:43.295 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:17:43.296 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:17:43.296 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:17:43.296 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:17:43.296 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:17:43.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:17:43.576 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:17:43.579 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:17:43.579 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:17:43.579 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:17:43.579 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:17:43.579 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:17:43.580 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:17:43.580 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:17:43.580 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:17:43.581 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:17:43.581 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:17:43.581 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:17:43.581 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:17:43.582 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:17:43.582 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:17:43.680 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:17:43.680 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:17:43.680 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:17:43.681 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:17:43.681 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:17:43.681 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:17:43.682 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:17:43.682 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:17:43.682 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:17:43.682 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:17:43.682 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:17:43.682 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:17:43.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:17:43.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:17:43.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2023-09-30T01:17:43.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:17:43.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:17:43.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:17:43.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:17:43.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:17:43.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:17:44.673 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:17:44.673 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:17:44.673 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:17:44.674 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:17:44.674 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:17:44.674 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:17:44.674 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:17:44.674 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:17:44.675 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:17:44.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2023-09-30T01:17:44.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:17:44.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:17:44.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:17:44.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:17:44.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:17:44.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:17:44.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:17:44.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:17:44.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:17:44.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:17:44.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:17:44.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:17:44.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:17:44.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:17:44.710 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:17:44.710 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:17:44.710 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:17:44.710 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:17:44.711 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:17:44.711 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:17:44.711 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:17:44.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:17:44.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:17:44.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:17:44.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:17:44.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:17:44.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:17:44.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:17:44.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:17:44.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:17:44.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:17:44.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:17:44.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:17:44.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:17:44.713 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:17:44.714 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8e9d5d38-0a30-43c2-a19d-33f60634325e 2023-09-30T01:17:44.714 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8e9d5d38-0a30-43c2-a19d-33f60634325e' 2023-09-30T01:17:44.714 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 8e9d5d38-0a30-43c2-a19d-33f60634325e 2023-09-30T01:17:44.714 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:17:44.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4dxdlzAyIKxAAtyt+qQq6GOJGu9G3IyvLtQ== 2023-09-30T01:17:44.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4dxdlzAyIKxAAtyt+qQq6GOJGu9G3IyvLtQ=="}' 2023-09-30T01:17:44.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8e9d5d38-0a30-43c2-a19d-33f60634325e -i td/ceph-helpers/0/new.json 2023-09-30T01:17:45.103 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:17:45.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:17:45.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4dxdlzAyIKxAAtyt+qQq6GOJGu9G3IyvLtQ== --osd-uuid 8e9d5d38-0a30-43c2-a19d-33f60634325e 2023-09-30T01:17:45.150 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:45.147+0000 7fc1d7df03c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:45.153 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:45.147+0000 7fc1d7df03c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:45.154 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:45.151+0000 7fc1d7df03c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:45.156 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:45.151+0000 7fc1d7df03c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:17:45.156 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:45.151+0000 7fc1d7df03c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:17:45.156 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:45.151+0000 7fc1d7df03c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:17:45.234 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:45.231+0000 7fc1d7df03c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:17:51.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:17:51.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:17:51.831 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:17:51.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:17:51.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:17:52.440 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:17:52.441 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:17:52.441 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:17:52.441 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:17:52.441 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:17:52.442 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:17:52.472 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:52.467+0000 7fa71f5bd3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:52.483 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:52.479+0000 7fa71f5bd3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:52.485 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:52.479+0000 7fa71f5bd3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:17:52.896 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:17:52.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:17:52.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:17:52.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:17:52.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:17:52.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:52.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:17:52.897 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:17:52.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:52.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:17:53.040 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:53.035+0000 7fa71f5bd3c0 -1 Falling back to public interface 2023-09-30T01:17:53.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:54.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:54.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:54.337 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:17:54.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:17:54.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:54.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:17:54.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:55.536 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:17:55.531+0000 7fa71f5bd3c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:17:55.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:55.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:55.813 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:17:55.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:17:55.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:55.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:17:56.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:57.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:57.313 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:57.313 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:17:57.313 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:17:57.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:57.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:17:57.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:17:58.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:17:58.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:17:58.734 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:17:58.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:17:58.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:17:58.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:17:59.207 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:00.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:00.209 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:18:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:18:00.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:00.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:00.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:01.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:01.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:01.641 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:18:01.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:18:01.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:01.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:02.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:03.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:03.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:03.047 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:18:03.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:18:03.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:03.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:03.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:04.568 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:04.568 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:04.569 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:18:04.569 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:18:04.569 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:04.569 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:05.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:06.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:06.020 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:06.020 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:18:06.020 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:18:06.020 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:06.021 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:07.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:08.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:08.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:08.012 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:18:08.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:18:08.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:08.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:09.123 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:10.125 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:10.125 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:10.125 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:18:10.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:18:10.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:10.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:11.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:12.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:12.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:12.319 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:18:12.320 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:18:12.320 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:12.320 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:13.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:14.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:14.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:14.446 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:18:14.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:18:14.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:14.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:15.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:16.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:16.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:16.135 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:18:16.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:18:16.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:16.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:16.888 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:17.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:17.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:17.890 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:18:17.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:18:17.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:17.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:18.337 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/77925,v1:127.0.0.1:6803/77925] [v2:127.0.0.1:6804/77925,v1:127.0.0.1:6805/77925] exists,up 8e9d5d38-0a30-43c2-a19d-33f60634325e 2023-09-30T01:18:18.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:18:18.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:18:18.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:18:18.339 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2023-09-30T01:18:18.339 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:18:18.339 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:18:18.340 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:18:18.340 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:18.340 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:18.340 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.0.asok 2023-09-30T01:18:18.341 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2023-09-30T01:18:18.341 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-osd.0.asok config get osd_max_backfills 2023-09-30T01:18:18.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2023-09-30T01:18:18.432 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2023-09-30T01:18:18.432 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2023-09-30T01:18:18.434 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2023-09-30T01:18:18.435 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:18:18.435 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:18:18.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:18:18.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:18:18.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:18:19.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:18:19.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2023-09-30T01:18:19.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2023-09-30T01:18:19.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-09-30T01:18:19.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-09-30T01:18:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-09-30T01:18:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:18:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:18:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:18:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:18:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:18:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:18:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:18:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-09-30T01:18:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-09-30T01:18:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:18:19.752 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-09-30T01:18:19.752 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:18:19.752 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:18:19.752 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:18:19.752 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:19.752 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:19.753 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:18:19.753 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:18:19.753 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:18:19.753 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:18:19.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:18:19.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:18:19.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:18:19.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:18:19.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:18:19.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-09-30T01:18:19.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2023-09-30T01:18:19.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:18:19.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-09-30T01:18:19.755 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:18:19.755 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2023-09-30T01:18:19.755 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2023-09-30T01:18:19.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-09-30T01:18:19.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-09-30T01:18:19.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-09-30T01:18:19.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-09-30T01:18:19.787 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:19.783+0000 7fa6a497d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:18:19.790 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:19.787+0000 7fa6a497d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:18:19.792 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:19.787+0000 7fa6a497d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:18:20.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-09-30T01:18:20.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:18:20.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:18:20.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:18:20.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:18:20.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:20.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:18:20.164 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:18:20.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:20.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:20.339 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:20.335+0000 7fa6a497d3c0 -1 Falling back to public interface 2023-09-30T01:18:20.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:21.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:21.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:21.634 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:18:21.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:18:21.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:21.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:22.043 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:23.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:23.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:23.046 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:18:23.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:18:23.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:23.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:23.206 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:23.203+0000 7fa6a497d3c0 -1 osd.0 6 log_to_monitors true 2023-09-30T01:18:23.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:24.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:24.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:24.497 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:18:24.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:18:24.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:24.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:24.905 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/78837,v1:127.0.0.1:6803/78837] [v2:127.0.0.1:6804/78837,v1:127.0.0.1:6805/78837] exists,up 8e9d5d38-0a30-43c2-a19d-33f60634325e 2023-09-30T01:18:24.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:18:24.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:18:24.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:18:24.906 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2023-09-30T01:18:24.906 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:18:24.906 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:18:24.907 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:18:24.907 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:24.907 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:24.907 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.0.asok 2023-09-30T01:18:24.908 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2023-09-30T01:18:24.908 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-osd.0.asok config get osd_max_backfills 2023-09-30T01:18:24.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2023-09-30T01:18:24.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2023-09-30T01:18:24.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2023-09-30T01:18:24.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:18:24.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:18:24.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:18:24.988 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:18:24.988 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:18:24.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:18:24.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:18:24.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:18:25.103 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:18:25.104 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:18:25.105 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:18:25.106 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:18:25.108 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:18:25.108 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:18:25.108 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:18:25.110 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:18:25.110 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:18:25.111 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:18:25.111 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:18:25.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:18:25.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:18:25.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:18:25.126 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:18:25.126 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:25.126 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:25.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:18:25.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:18:25.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:18:25.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:18:25.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2023-09-30T01:18:25.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2023-09-30T01:18:25.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2023-09-30T01:18:25.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:18:25.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:18:25.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:18:25.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:18:25.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:18:25.129 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:18:25.129 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:18:25.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:18:25.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:18:25.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:18:25.132 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:18:25.133 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:18:25.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:18:25.135 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:18:25.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:18:25.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:18:25.138 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:18:25.138 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:18:25.139 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:18:25.139 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:18:25.139 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:18:25.141 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:18:25.142 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:18:25.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:18:25.145 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:18:25.145 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:25.145 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:25.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:18:25.147 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:18:25.147 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:18:25.147 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:18:25.149 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:18:25.149 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:25.149 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:25.149 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:18:25.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:18:25.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:18:25.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:18:25.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:18:25.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2023-09-30T01:18:25.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:18:25.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:18:25.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:18:25.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:18:25.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:18:25.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2023-09-30T01:18:25.521 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:18:25.522 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:18:25.522 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:18:25.522 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:18:25.522 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:25.522 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:25.523 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:18:25.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:18:25.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:18:25.833 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:18:25.833 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:18:25.833 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:18:25.833 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:18:25.834 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:18:25.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:18:25.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:18:25.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:18:25.835 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:18:25.835 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:25.835 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:25.835 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:18:25.835 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:18:25.835 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:18:25.924 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:18:25.924 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:18:25.924 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:18:25.924 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:18:25.924 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:18:25.925 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:18:25.925 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:18:25.925 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:18:25.925 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:18:25.926 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:25.926 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:25.926 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:18:25.926 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:18:25.927 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:18:26.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2023-09-30T01:18:26.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:18:26.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:18:26.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:18:26.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:18:26.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:18:26.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:18:26.909 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:18:26.909 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:18:26.909 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:18:26.910 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:18:26.910 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:26.911 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:26.911 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:18:26.911 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:18:26.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:18:26.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2023-09-30T01:18:26.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:18:26.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:18:26.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:18:26.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:18:26.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:18:26.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:18:26.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:18:26.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:18:26.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:18:26.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:18:26.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:18:26.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:18:26.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:18:26.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:18:26.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:18:26.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:18:26.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:18:26.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:18:26.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:18:26.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:18:26.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:18:26.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:18:26.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:18:26.950 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:18:26.950 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dedc85d2-1703-4417-ba5d-11463b0b0f45 2023-09-30T01:18:26.950 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 dedc85d2-1703-4417-ba5d-11463b0b0f45 2023-09-30T01:18:26.951 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dedc85d2-1703-4417-ba5d-11463b0b0f45' 2023-09-30T01:18:26.951 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:18:26.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBidxdlGY6cORAAaqLjakOehmhHIHuPN4u6dA== 2023-09-30T01:18:26.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBidxdlGY6cORAAaqLjakOehmhHIHuPN4u6dA=="}' 2023-09-30T01:18:26.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dedc85d2-1703-4417-ba5d-11463b0b0f45 -i td/ceph-helpers/0/new.json 2023-09-30T01:18:27.284 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:18:27.295 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:18:27.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBidxdlGY6cORAAaqLjakOehmhHIHuPN4u6dA== --osd-uuid dedc85d2-1703-4417-ba5d-11463b0b0f45 2023-09-30T01:18:27.329 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:27.322+0000 7f96eac2c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:18:27.331 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:27.326+0000 7f96eac2c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:18:27.332 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:27.326+0000 7f96eac2c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:18:27.333 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:27.330+0000 7f96eac2c3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:18:27.333 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:27.330+0000 7f96eac2c3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:18:27.333 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:27.330+0000 7f96eac2c3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:18:27.377 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:27.370+0000 7f96eac2c3c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:18:33.574 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:18:33.575 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:18:33.577 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:18:33.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:18:33.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:18:34.127 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:18:34.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:18:34.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:18:34.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:18:34.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:18:34.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:18:34.156 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:34.150+0000 7f53304373c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:18:34.159 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:34.154+0000 7f53304373c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:18:34.160 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:34.154+0000 7f53304373c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:18:34.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:18:34.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:18:34.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:18:34.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:18:34.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:18:34.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:34.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:18:34.566 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:18:34.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:34.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:34.708 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:34.702+0000 7f53304373c0 -1 Falling back to public interface 2023-09-30T01:18:34.995 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:35.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:35.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:35.998 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:18:35.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:18:35.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:35.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:36.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:37.118 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:18:37.114+0000 7f53304373c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:18:37.398 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:37.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:37.399 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:18:37.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:18:37.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:37.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:37.928 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:38.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:38.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:38.930 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:18:38.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:18:38.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:38.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:39.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:40.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:40.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:40.375 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:18:40.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:18:40.376 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:40.376 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:40.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:41.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:41.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:41.811 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:18:41.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:18:41.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:41.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:42.220 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:43.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:43.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:43.222 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:18:43.223 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:18:43.223 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:43.223 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:43.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:44.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:44.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:44.642 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:18:44.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:18:44.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:44.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:45.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:46.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:46.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:46.122 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:18:46.122 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:18:46.122 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:46.122 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:46.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:47.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:47.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:47.581 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:18:47.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:18:47.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:47.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:48.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:49.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:49.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:49.598 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:18:49.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:18:49.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:49.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:50.171 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:51.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:51.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:51.173 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:18:51.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:18:51.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:51.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:51.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:52.720 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:52.720 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:52.721 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:18:52.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:18:52.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:52.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:53.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:54.742 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:54.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:54.743 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:18:54.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:18:54.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:54.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:55.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:56.499 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:56.499 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:56.499 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:18:56.500 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:18:56.500 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:56.501 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:56.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:57.950 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:57.950 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:57.950 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:18:57.951 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:18:57.951 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:57.951 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:58.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:18:59.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:18:59.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:18:59.541 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:18:59.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:18:59.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:18:59.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:18:59.948 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/80547,v1:127.0.0.1:6803/80547] [v2:127.0.0.1:6804/80547,v1:127.0.0.1:6805/80547] exists,up dedc85d2-1703-4417-ba5d-11463b0b0f45 2023-09-30T01:18:59.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:18:59.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:18:59.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:18:59.950 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2023-09-30T01:18:59.950 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:18:59.950 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:18:59.950 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:18:59.951 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:18:59.951 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:18:59.951 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.0.asok 2023-09-30T01:18:59.951 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2023-09-30T01:18:59.952 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-osd.0.asok config get osd_max_backfills 2023-09-30T01:19:00.038 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2023-09-30T01:19:00.039 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2023-09-30T01:19:00.039 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2023-09-30T01:19:00.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2023-09-30T01:19:00.042 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:19:00.042 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:19:00.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:19:00.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:19:00.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:19:00.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:19:00.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2023-09-30T01:19:00.855 INFO:tasks.workunit.client.0.smithi186.stderr:osd.0 is already down. 2023-09-30T01:19:00.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2023-09-30T01:19:00.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2023-09-30T01:19:00.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:19:00.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:19:00.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:19:00.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:00.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:19:00.869 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:19:00.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:00.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2023-09-30T01:19:01.277 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/80547,v1:127.0.0.1:6803/80547] [v2:127.0.0.1:6804/80547,v1:127.0.0.1:6805/80547] exists dedc85d2-1703-4417-ba5d-11463b0b0f45 2023-09-30T01:19:01.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:19:01.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:19:01.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:19:01.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2023-09-30T01:19:01.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2023-09-30T01:19:01.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-09-30T01:19:01.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-09-30T01:19:01.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-09-30T01:19:01.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:19:01.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:19:01.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:19:01.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:19:01.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:19:01.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:19:01.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:19:01.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-09-30T01:19:01.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-09-30T01:19:01.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:19:01.281 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-09-30T01:19:01.281 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:19:01.281 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:19:01.282 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:19:01.282 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:01.282 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:01.282 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:01.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:01.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:19:01.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:19:01.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:19:01.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:19:01.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:19:01.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:19:01.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:19:01.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-09-30T01:19:01.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2023-09-30T01:19:01.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:19:01.284 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:19:01.285 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-09-30T01:19:01.285 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2023-09-30T01:19:01.285 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2023-09-30T01:19:01.285 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-09-30T01:19:01.285 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-09-30T01:19:01.286 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-09-30T01:19:01.286 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-09-30T01:19:01.312 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:01.306+0000 7f78c7e803c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:01.326 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:01.322+0000 7f78c7e803c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:01.327 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:01.322+0000 7f78c7e803c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:01.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-09-30T01:19:01.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:19:01.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:19:01.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:19:01.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:19:01.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:01.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:19:01.684 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:19:01.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:01.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:01.875 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:01.870+0000 7f78c7e803c0 -1 Falling back to public interface 2023-09-30T01:19:02.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:03.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:03.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:03.145 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:19:03.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:19:03.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:03.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:03.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:04.609 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:04.610 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:04.610 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:19:04.610 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:19:04.610 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:04.610 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:04.832 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:04.826+0000 7f78c7e803c0 -1 osd.0 5 log_to_monitors true 2023-09-30T01:19:05.105 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:06.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:06.107 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:06.107 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:19:06.107 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:19:06.108 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:06.108 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:06.564 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/81533,v1:127.0.0.1:6803/81533] [v2:127.0.0.1:6804/81533,v1:127.0.0.1:6805/81533] exists,up dedc85d2-1703-4417-ba5d-11463b0b0f45 2023-09-30T01:19:06.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:19:06.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:19:06.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:19:06.566 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2023-09-30T01:19:06.566 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:19:06.566 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:19:06.567 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:19:06.567 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:06.567 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:06.567 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.0.asok 2023-09-30T01:19:06.568 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2023-09-30T01:19:06.568 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-osd.0.asok config get osd_max_backfills 2023-09-30T01:19:06.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2023-09-30T01:19:06.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2023-09-30T01:19:06.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2023-09-30T01:19:06.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:19:06.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:19:06.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:19:06.658 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:19:06.658 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:19:06.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:19:06.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:19:06.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:19:06.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:19:06.772 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:19:06.773 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:19:06.774 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:19:06.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:19:06.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:19:06.776 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:19:06.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:19:06.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:19:06.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:19:06.780 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:19:06.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:19:06.783 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:19:06.783 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:19:06.793 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:19:06.794 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:06.794 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:06.794 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:19:06.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:19:06.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:19:06.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:19:06.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2023-09-30T01:19:06.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2023-09-30T01:19:06.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2023-09-30T01:19:06.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:19:06.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:19:06.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:19:06.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:19:06.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:19:06.799 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:19:06.799 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:19:06.799 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:19:06.799 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:19:06.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:19:06.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:19:06.802 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:19:06.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:19:06.803 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:19:06.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:19:06.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:19:06.806 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:19:06.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:19:06.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:19:06.809 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:19:06.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:19:06.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:19:06.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:19:06.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:19:06.813 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:19:06.813 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:06.813 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:06.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:19:06.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:19:06.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:19:06.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:19:06.817 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:19:06.817 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:06.817 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:06.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:19:06.820 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:19:06.821 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:19:06.821 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:19:06.821 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:19:06.821 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2023-09-30T01:19:06.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:19:06.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:19:06.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:19:06.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:19:06.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:19:06.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2023-09-30T01:19:07.222 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:19:07.222 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:19:07.222 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:19:07.223 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:19:07.223 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:07.223 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:07.223 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:07.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:19:07.505 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:19:07.506 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:19:07.506 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:19:07.507 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:19:07.507 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:19:07.507 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:19:07.508 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:19:07.508 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:19:07.508 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:19:07.509 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:19:07.509 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:07.509 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:07.509 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:19:07.510 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:19:07.511 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:19:07.607 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:19:07.607 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:19:07.607 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:19:07.607 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:19:07.607 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:19:07.608 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:19:07.608 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:19:07.608 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:19:07.608 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:19:07.608 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:07.609 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:07.609 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:19:07.609 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:19:07.609 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:19:07.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2023-09-30T01:19:07.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:19:07.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:19:07.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:19:07.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:19:07.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:19:07.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:19:08.617 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:19:08.617 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:19:08.617 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:19:08.618 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:19:08.618 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:08.618 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:08.618 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:08.619 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:19:08.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:19:08.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2023-09-30T01:19:08.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:19:08.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:19:08.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:19:08.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:19:08.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:19:08.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:19:08.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:19:08.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:19:08.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:19:08.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:19:08.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:19:08.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:19:08.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:19:08.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:19:08.658 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:19:08.658 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:19:08.658 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:19:08.658 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:19:08.659 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:08.659 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:08.659 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:08.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:08.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:19:08.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:19:08.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:19:08.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:19:08.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:19:08.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:19:08.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:19:08.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:19:08.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:19:08.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:19:08.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2023-09-30T01:19:08.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:19:08.662 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:19:08.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c1319ed0-3fb8-43b9-9a1a-56bbea7e6240 2023-09-30T01:19:08.662 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 c1319ed0-3fb8-43b9-9a1a-56bbea7e6240 2023-09-30T01:19:08.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c1319ed0-3fb8-43b9-9a1a-56bbea7e6240' 2023-09-30T01:19:08.663 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:19:08.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMdxdlWuarKBAAulxNw92WsUMsuOHmGPW9PQ== 2023-09-30T01:19:08.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMdxdlWuarKBAAulxNw92WsUMsuOHmGPW9PQ=="}' 2023-09-30T01:19:08.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c1319ed0-3fb8-43b9-9a1a-56bbea7e6240 -i td/ceph-helpers/0/new.json 2023-09-30T01:19:09.072 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:19:09.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:19:09.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQCMdxdlWuarKBAAulxNw92WsUMsuOHmGPW9PQ== --osd-uuid c1319ed0-3fb8-43b9-9a1a-56bbea7e6240 2023-09-30T01:19:09.119 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:09.114+0000 7f622e3ba3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:09.122 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:09.118+0000 7f622e3ba3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:09.123 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:09.118+0000 7f622e3ba3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:09.125 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:09.118+0000 7f622e3ba3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:19:09.125 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:09.118+0000 7f622e3ba3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:19:09.125 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:09.118+0000 7f622e3ba3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:19:09.203 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:09.198+0000 7f622e3ba3c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:19:15.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:19:15.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:19:15.432 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:19:15.432 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:19:15.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:19:15.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:19:15.946 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:19:15.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2023-09-30T01:19:15.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:19:15.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:19:15.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:19:15.979 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:15.974+0000 7fd506e153c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:15.981 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:15.978+0000 7fd506e153c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:15.986 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:15.982+0000 7fd506e153c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:16.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:19:16.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:19:16.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:19:16.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:19:16.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:19:16.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:16.395 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:19:16.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:19:16.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:16.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:16.567 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:16.562+0000 7fd506e153c0 -1 Falling back to public interface 2023-09-30T01:19:16.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:17.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:17.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:17.818 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:19:17.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:19:17.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:17.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:18.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:19.060 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:19.054+0000 7fd506e153c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:19:19.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:19.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:19.277 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:19:19.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:19:19.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:19.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:20.753 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:20.753 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:20.754 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:19:20.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:19:20.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:20.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:21.135 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/83243,v1:127.0.0.1:6803/83243] [v2:127.0.0.1:6804/83243,v1:127.0.0.1:6805/83243] exists,up c1319ed0-3fb8-43b9-9a1a-56bbea7e6240 2023-09-30T01:19:21.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:19:21.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:19:21.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:19:21.136 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2023-09-30T01:19:21.137 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:19:21.137 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:19:21.137 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:19:21.137 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:21.138 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:21.138 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.0.asok 2023-09-30T01:19:21.138 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2023-09-30T01:19:21.138 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2023-09-30T01:19:21.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2023-09-30T01:19:21.228 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2023-09-30T01:19:21.228 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:19:21.228 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:19:21.229 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:19:21.229 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:21.229 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:21.230 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.0.asok 2023-09-30T01:19:21.230 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2023-09-30T01:19:21.230 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2023-09-30T01:19:21.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2023-09-30T01:19:21.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2023-09-30T01:19:21.322 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:19:21.323 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:19:21.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:19:21.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:19:21.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:19:21.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:19:21.636 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2023-09-30T01:19:22.093 INFO:tasks.workunit.client.0.smithi186.stderr:osd.0 is already down. 2023-09-30T01:19:22.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2023-09-30T01:19:22.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2023-09-30T01:19:22.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:19:22.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:19:22.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:19:22.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:22.101 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:19:22.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:19:22.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:22.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2023-09-30T01:19:22.504 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/83243,v1:127.0.0.1:6803/83243] [v2:127.0.0.1:6804/83243,v1:127.0.0.1:6805/83243] exists c1319ed0-3fb8-43b9-9a1a-56bbea7e6240 2023-09-30T01:19:22.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:19:22.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:19:22.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:19:22.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2023-09-30T01:19:22.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2023-09-30T01:19:22.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-09-30T01:19:22.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-09-30T01:19:22.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-09-30T01:19:22.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:19:22.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:19:22.509 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:19:22.510 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:22.510 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:22.510 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:22.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:22.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:19:22.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:19:22.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:19:22.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:19:22.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:19:22.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:19:22.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:19:22.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-09-30T01:19:22.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2023-09-30T01:19:22.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:19:22.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-09-30T01:19:22.511 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:19:22.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2023-09-30T01:19:22.512 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2023-09-30T01:19:22.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-09-30T01:19:22.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-09-30T01:19:22.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-09-30T01:19:22.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-09-30T01:19:22.536 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:22.530+0000 7f77525ff3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:22.548 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:22.542+0000 7f77525ff3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:22.550 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:22.546+0000 7f77525ff3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:22.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-09-30T01:19:22.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:19:22.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:19:22.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:19:22.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:19:22.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:22.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:19:22.955 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:19:22.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:22.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:23.102 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:23.098+0000 7f77525ff3c0 -1 Falling back to public interface 2023-09-30T01:19:23.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:24.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:24.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:24.372 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:19:24.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:19:24.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:24.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:24.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:25.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:25.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:25.766 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:19:25.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:19:25.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:25.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:25.936 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:25.930+0000 7f77525ff3c0 -1 osd.0 6 log_to_monitors true 2023-09-30T01:19:26.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:27.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:27.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:27.203 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:19:27.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:19:27.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:27.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:27.609 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/83893,v1:127.0.0.1:6803/83893] [v2:127.0.0.1:6804/83893,v1:127.0.0.1:6805/83893] exists,up c1319ed0-3fb8-43b9-9a1a-56bbea7e6240 2023-09-30T01:19:27.609 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:19:27.609 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:19:27.609 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:19:27.610 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2023-09-30T01:19:27.610 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:19:27.610 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:19:27.611 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:19:27.611 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:27.611 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:27.611 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.0.asok 2023-09-30T01:19:27.612 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2023-09-30T01:19:27.612 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2023-09-30T01:19:27.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2023-09-30T01:19:27.700 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2023-09-30T01:19:27.701 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:19:27.701 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:19:27.701 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:19:27.701 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:27.702 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:27.702 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.0.asok 2023-09-30T01:19:27.702 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2023-09-30T01:19:27.702 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2023-09-30T01:19:27.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2023-09-30T01:19:27.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2023-09-30T01:19:27.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2023-09-30T01:19:27.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2023-09-30T01:19:27.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:19:27.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:19:27.794 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:19:27.794 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:19:27.794 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:19:27.794 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:19:27.794 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:19:27.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:19:27.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:19:27.907 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:19:27.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:19:27.909 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:19:27.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:19:27.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:19:27.912 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:19:27.914 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:19:27.914 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:19:27.914 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:19:27.915 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:19:27.916 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:19:27.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:19:27.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:19:27.925 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:19:27.925 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:27.925 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:27.926 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:19:27.928 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_destroy_osd td/ceph-helpers 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:19:27.929 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:19:27.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:19:27.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:19:27.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:19:27.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:19:27.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:19:27.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:19:27.935 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:19:27.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:19:27.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:19:27.937 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:19:27.939 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:19:27.939 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:19:27.939 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:19:27.940 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:19:27.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:19:27.943 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:19:27.943 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:19:27.945 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:19:27.945 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:27.946 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:27.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:19:27.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:19:27.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:19:27.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:19:27.950 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:19:27.950 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:27.950 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:27.950 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:19:27.953 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:19:27.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:19:27.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:19:27.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:19:27.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2023-09-30T01:19:27.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:19:27.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:19:27.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:19:27.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:19:27.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:19:27.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2023-09-30T01:19:28.370 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:19:28.370 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:19:28.370 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:19:28.371 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:19:28.371 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:28.371 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:28.371 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:28.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:19:28.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:19:28.661 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:19:28.661 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:19:28.661 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:19:28.661 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:19:28.662 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:19:28.662 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:19:28.662 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:19:28.662 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:19:28.663 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:19:28.663 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:28.663 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:28.663 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:19:28.664 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:19:28.664 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:19:28.757 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:19:28.758 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:19:28.758 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:19:28.758 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:19:28.759 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:19:28.759 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:19:28.759 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:19:28.759 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:19:28.759 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:19:28.759 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:28.759 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:28.760 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:19:28.760 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:19:28.760 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:19:28.846 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2023-09-30T01:19:28.846 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:19:28.846 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:19:28.846 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:19:28.846 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:19:28.846 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:19:28.847 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:19:29.752 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:19:29.752 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:19:29.752 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:19:29.752 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:19:29.752 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:29.752 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:29.752 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:29.753 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:19:29.755 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:19:29.784 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2023-09-30T01:19:29.784 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:19:29.784 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:19:29.784 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:19:29.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:19:29.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:19:29.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:19:29.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:19:29.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:19:29.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:19:29.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:19:29.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:19:29.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:19:29.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:19:29.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:19:29.786 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:19:29.786 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:19:29.786 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:19:29.787 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:19:29.788 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:19:29.788 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:19:29.788 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:29.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:19:29.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:19:29.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:19:29.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:19:29.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:19:29.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:19:29.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:19:29.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:19:29.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:19:29.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:19:29.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:19:29.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:19:29.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:19:29.790 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:19:29.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8b1ff34c-b0f0-40d2-856d-4db2f3619245 2023-09-30T01:19:29.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8b1ff34c-b0f0-40d2-856d-4db2f3619245' 2023-09-30T01:19:29.792 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 8b1ff34c-b0f0-40d2-856d-4db2f3619245 2023-09-30T01:19:29.792 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:19:29.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQChdxdlLYwxMBAA2/Ar3eDLyY63zLQ54Zsrxw== 2023-09-30T01:19:29.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQChdxdlLYwxMBAA2/Ar3eDLyY63zLQ54Zsrxw=="}' 2023-09-30T01:19:29.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8b1ff34c-b0f0-40d2-856d-4db2f3619245 -i td/ceph-helpers/0/new.json 2023-09-30T01:19:30.186 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:19:30.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:19:30.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChdxdlLYwxMBAA2/Ar3eDLyY63zLQ54Zsrxw== --osd-uuid 8b1ff34c-b0f0-40d2-856d-4db2f3619245 2023-09-30T01:19:30.236 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:30.230+0000 7f98176603c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:30.239 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:30.234+0000 7f98176603c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:30.241 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:30.238+0000 7f98176603c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:30.241 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:30.238+0000 7f98176603c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:19:30.242 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:30.238+0000 7f98176603c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:19:30.242 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:30.238+0000 7f98176603c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:19:30.317 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:30.314+0000 7f98176603c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:19:36.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:19:36.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:19:36.546 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:19:36.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:19:36.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:19:37.112 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:19:37.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:19:37.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:19:37.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:19:37.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:19:37.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:19:37.139 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:37.134+0000 7f98c5d313c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:37.143 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:37.138+0000 7f98c5d313c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:37.149 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:37.138+0000 7f98c5d313c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:19:37.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:19:37.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:19:37.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:19:37.566 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:19:37.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:19:37.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:19:37.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:37.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:19:37.567 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:37.567 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:37.700 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:37.694+0000 7f98c5d313c0 -1 Falling back to public interface 2023-09-30T01:19:38.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:39.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:39.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:39.012 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:19:39.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:19:39.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:39.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:39.423 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:40.186 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:19:40.182+0000 7f98c5d313c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:19:40.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:40.425 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:40.425 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:19:40.425 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:19:40.426 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:40.426 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:40.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:41.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:41.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:41.989 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:19:41.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:19:41.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:41.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:42.421 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:43.423 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:43.423 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:43.424 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:19:43.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:19:43.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:43.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:43.921 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:44.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:44.924 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:44.924 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:19:44.924 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:19:44.924 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:44.925 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:45.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:46.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:46.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:46.342 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:19:46.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:19:46.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:46.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:46.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:47.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:47.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:47.903 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:19:47.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:19:47.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:47.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:48.364 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:49.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:49.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:49.366 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:19:49.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:19:49.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:49.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:49.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:50.807 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:50.807 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:50.807 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:19:50.807 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:19:50.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:50.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:51.471 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:52.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:52.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:52.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:19:52.473 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:19:52.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:52.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:53.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:54.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:54.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:54.627 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:19:54.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:19:54.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:54.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:55.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:56.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:56.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:56.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:19:56.659 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:19:56.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:56.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:57.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:19:58.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:19:58.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:19:58.658 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:19:58.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:19:58.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:19:58.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:19:59.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:00.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:00.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:00.424 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:20:00.425 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:20:00.425 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:00.425 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:01.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:02.233 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:02.233 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:02.233 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:20:02.234 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:20:02.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:02.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:02.734 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85607,v1:127.0.0.1:6803/85607] [v2:127.0.0.1:6804/85607,v1:127.0.0.1:6805/85607] exists,up 8b1ff34c-b0f0-40d2-856d-4db2f3619245 2023-09-30T01:20:02.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:20:02.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:20:02.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:20:02.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2023-09-30T01:20:02.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2023-09-30T01:20:02.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2023-09-30T01:20:02.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2023-09-30T01:20:03.402 INFO:tasks.workunit.client.0.smithi186.stderr:osd.0 is already out. 2023-09-30T01:20:03.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2023-09-30T01:20:03.416 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:03.416 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:03.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:03.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:03.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:03.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:03.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2023-09-30T01:20:04.194 INFO:tasks.workunit.client.0.smithi186.stderr:osd.0 is already down. 2023-09-30T01:20:04.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2023-09-30T01:20:04.729 INFO:tasks.workunit.client.0.smithi186.stderr:osd.0 does not exist 2023-09-30T01:20:04.742 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/ceph-helpers/0 2023-09-30T01:20:04.742 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers/0 2023-09-30T01:20:04.742 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:20:04.742 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers/0 KILL 2023-09-30T01:20:04.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:04.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:04.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:04.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:04.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:04.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:04.747 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:20:04.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:20:04.749 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:20:04.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:20:04.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:20:04.750 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:20:04.752 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:04.752 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:20:04.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:20:04.757 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:04.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:20:04.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:20:04.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers/0 2023-09-30T01:20:04.762 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:20:04.762 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:04.762 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:04.762 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:20:04.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:20:04.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:20:04.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/ceph-helpers/0 2023-09-30T01:20:04.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: ceph osd dump 2023-09-30T01:20:04.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: grep 'osd. ' 2023-09-30T01:20:05.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:804: test_destroy_osd: teardown td/ceph-helpers 2023-09-30T01:20:05.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:20:05.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:20:05.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:20:05.258 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:05.258 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:05.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:05.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:05.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:05.371 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:05.371 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:20:05.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:20:05.373 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:20:05.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:20:05.376 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:20:05.376 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:20:05.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:05.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:20:05.379 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:20:05.379 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:05.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:20:05.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:20:05.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:20:05.389 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:20:05.390 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:05.390 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:05.390 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:20:05.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:20:05.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:20:05.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:20:05.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_display_logs td/ceph-helpers 2023-09-30T01:20:05.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: test_display_logs: local dir=td/ceph-helpers 2023-09-30T01:20:05.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2036: test_display_logs: setup td/ceph-helpers 2023-09-30T01:20:05.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:20:05.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:20:05.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:20:05.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:20:05.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:20:05.395 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:05.395 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:05.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:05.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:05.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:05.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:05.398 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:20:05.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:20:05.400 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:20:05.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:20:05.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:20:05.403 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:20:05.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:05.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:20:05.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:20:05.405 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:05.406 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:20:05.408 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:20:05.408 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:20:05.410 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:20:05.411 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:05.411 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:05.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:20:05.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:20:05.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:20:05.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:20:05.415 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:20:05.415 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:05.415 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:05.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:20:05.418 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:20:05.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:20:05.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:20:05.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:20:05.420 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: test_display_logs: run_mon td/ceph-helpers a 2023-09-30T01:20:05.420 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:20:05.420 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:20:05.420 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:20:05.420 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:20:05.420 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:20:05.420 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2023-09-30T01:20:05.830 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:20:05.830 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:20:05.830 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:20:05.830 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:20:05.831 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:05.831 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:05.831 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:20:05.831 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:20:06.120 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:20:06.121 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:20:06.122 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:20:06.122 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:20:06.122 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:20:06.122 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:20:06.123 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:20:06.123 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:20:06.123 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:20:06.124 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:20:06.124 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:06.124 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:06.124 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:20:06.124 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:20:06.124 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:20:06.212 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:20:06.212 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:20:06.213 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:20:06.213 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:20:06.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:20:06.214 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:20:06.214 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:20:06.214 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:20:06.214 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:20:06.214 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:06.214 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:06.214 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:20:06.215 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:20:06.215 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:20:06.301 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2038: test_display_logs: kill_daemons td/ceph-helpers 2023-09-30T01:20:06.302 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:06.302 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:06.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:06.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:06.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:07.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:07.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: test_display_logs: display_logs td/ceph-helpers 2023-09-30T01:20:07.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: display_logs: local dir=td/ceph-helpers 2023-09-30T01:20:07.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2023-09-30T01:20:07.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: display_logs: read file 2023-09-30T01:20:07.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2023-09-30T01:20:07.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: display_logs: cat td/ceph-helpers/mon.a.log 2023-09-30T01:20:07.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2027: display_logs: read file 2023-09-30T01:20:07.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2023-09-30T01:20:07.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2041: test_display_logs: teardown td/ceph-helpers 2023-09-30T01:20:07.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:20:07.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:20:07.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:20:07.624 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:07.625 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:07.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:07.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:07.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:07.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:07.630 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:20:07.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:20:07.632 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:20:07.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:20:07.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:20:07.635 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:20:07.636 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:07.636 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:20:07.637 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:20:07.638 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:07.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:20:07.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:20:07.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:20:07.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:20:07.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:07.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:07.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:20:07.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:20:07.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:20:07.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:20:07.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2023-09-30T01:20:07.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2005: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2023-09-30T01:20:07.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2007: test_erasure_code_plugin_exists: setup td/ceph-helpers 2023-09-30T01:20:07.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:20:07.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:20:07.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:20:07.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:20:07.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:20:07.648 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:07.648 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:07.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:07.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:07.649 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:07.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:07.652 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:20:07.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:20:07.654 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:20:07.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:20:07.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:20:07.656 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:20:07.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:20:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:20:07.659 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:07.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:20:07.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:20:07.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:20:07.664 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:20:07.664 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:07.664 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:07.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:20:07.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:20:07.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:20:07.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:20:07.668 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:20:07.668 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:07.669 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:07.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:20:07.672 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:20:07.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:20:07.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:20:07.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:20:07.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2023-09-30T01:20:07.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:20:07.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:20:07.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:20:07.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:20:07.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:20:07.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2023-09-30T01:20:08.053 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:20:08.054 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:20:08.054 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:20:08.054 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:20:08.054 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:08.055 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:08.055 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:20:08.055 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:20:08.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:20:08.369 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:20:08.369 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:20:08.369 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:20:08.369 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:20:08.370 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:20:08.370 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:20:08.371 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:20:08.371 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:20:08.371 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:20:08.372 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:08.372 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:08.372 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:20:08.372 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:20:08.372 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:20:08.461 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:20:08.461 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:20:08.461 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:20:08.461 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:20:08.462 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:20:08.462 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:20:08.462 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:20:08.462 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:20:08.465 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:20:08.465 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:08.465 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:08.466 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:20:08.466 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:20:08.466 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:20:08.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2023-09-30T01:20:08.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:20:08.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:20:08.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:20:08.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:20:08.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:20:08.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:20:09.442 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:20:09.442 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:20:09.442 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:20:09.443 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:20:09.443 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:09.443 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:09.443 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:20:09.443 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:20:09.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:20:09.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2023-09-30T01:20:09.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: erasure_code_plugin_exists: local plugin=jerasure 2023-09-30T01:20:09.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: erasure_code_plugin_exists: local status 2023-09-30T01:20:09.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: erasure_code_plugin_exists: local grepstr 2023-09-30T01:20:09.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: erasure_code_plugin_exists: local s 2023-09-30T01:20:09.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: erasure_code_plugin_exists: case `uname` in 2023-09-30T01:20:09.476 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: erasure_code_plugin_exists: uname 2023-09-30T01:20:09.478 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2023-09-30T01:20:09.479 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2023-09-30T01:20:09.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: erasure_code_plugin_exists: s= 2023-09-30T01:20:09.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: erasure_code_plugin_exists: local status=0 2023-09-30T01:20:09.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2023-09-30T01:20:09.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2023-09-30T01:20:10.306 INFO:tasks.workunit.client.0.smithi186.stderr:erasure-code-profile TESTPROFILE does not exist 2023-09-30T01:20:10.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: erasure_code_plugin_exists: return 0 2023-09-30T01:20:10.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2023-09-30T01:20:10.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: erasure_code_plugin_exists: local plugin=FAKE 2023-09-30T01:20:10.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: erasure_code_plugin_exists: local status 2023-09-30T01:20:10.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: erasure_code_plugin_exists: local grepstr 2023-09-30T01:20:10.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: erasure_code_plugin_exists: local s 2023-09-30T01:20:10.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: erasure_code_plugin_exists: case `uname` in 2023-09-30T01:20:10.319 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: erasure_code_plugin_exists: uname 2023-09-30T01:20:10.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2023-09-30T01:20:10.320 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2023-09-30T01:20:10.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so): /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2023-09-30T01:20:10.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: erasure_code_plugin_exists: local status=5 2023-09-30T01:20:10.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2023-09-30T01:20:10.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so):' /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2023-09-30T01:20:10.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2023-09-30T01:20:10.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: erasure_code_plugin_exists: return 5 2023-09-30T01:20:10.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2012: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2023-09-30T01:20:10.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:20:10.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:20:10.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:20:10.544 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:10.544 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:10.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:10.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:10.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:10.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:10.654 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:20:10.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:20:10.657 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:20:10.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:20:10.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:20:10.659 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:20:10.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:10.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:20:10.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:20:10.662 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:10.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:20:10.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:20:10.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:20:10.669 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:20:10.669 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:10.669 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:10.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:20:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:20:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:20:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:20:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_expect_failure td/ceph-helpers 2023-09-30T01:20:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1915: test_expect_failure: local dir=td/ceph-helpers 2023-09-30T01:20:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: test_expect_failure: setup td/ceph-helpers 2023-09-30T01:20:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:20:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:20:10.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:20:10.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:20:10.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:20:10.673 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:10.673 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:10.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:10.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:10.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:10.676 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:10.676 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:20:10.677 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:20:10.678 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:20:10.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:20:10.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:20:10.680 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:20:10.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:10.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:20:10.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:20:10.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:10.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:20:10.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:20:10.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:20:10.688 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:20:10.688 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:10.688 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:10.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:20:10.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:20:10.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:20:10.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:20:10.692 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:20:10.692 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:10.692 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:10.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:20:10.695 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:20:10.697 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:20:10.697 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:20:10.697 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:20:10.697 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2023-09-30T01:20:10.697 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/ceph-helpers 2023-09-30T01:20:10.697 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2023-09-30T01:20:10.697 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local expected=FAIL 2023-09-30T01:20:10.697 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2023-09-30T01:20:10.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2023-09-30T01:20:10.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: bash -c 'echo FAIL ; exit 1' 2023-09-30T01:20:10.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2023-09-30T01:20:10.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2023-09-30T01:20:10.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2023-09-30T01:20:10.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1910: expect_failure: return 0 2023-09-30T01:20:10.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2023-09-30T01:20:10.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/ceph-helpers 2023-09-30T01:20:10.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2023-09-30T01:20:10.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local expected=FAIL 2023-09-30T01:20:10.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2023-09-30T01:20:10.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2023-09-30T01:20:10.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: bash -c 'echo FAIL ; exit 0' 2023-09-30T01:20:10.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1901: expect_failure: success=true 2023-09-30T01:20:10.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: true 2023-09-30T01:20:10.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: expect_failure: cat td/ceph-helpers/out 2023-09-30T01:20:10.705 INFO:tasks.workunit.client.0.smithi186.stderr:FAIL 2023-09-30T01:20:10.705 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: expect_failure: return 1 2023-09-30T01:20:10.706 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2023-09-30T01:20:10.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2023-09-30T01:20:10.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: expect_failure: local dir=td/ceph-helpers 2023-09-30T01:20:10.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: expect_failure: shift 2023-09-30T01:20:10.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: expect_failure: local expected=FAIL 2023-09-30T01:20:10.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: expect_failure: shift 2023-09-30T01:20:10.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: expect_failure: local success 2023-09-30T01:20:10.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2023-09-30T01:20:10.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: expect_failure: success=false 2023-09-30T01:20:10.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: false 2023-09-30T01:20:10.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1906: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2023-09-30T01:20:10.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1907: expect_failure: cat td/ceph-helpers/out 2023-09-30T01:20:10.713 INFO:tasks.workunit.client.0.smithi186.stderr:UNEXPECTED 2023-09-30T01:20:10.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1908: expect_failure: return 1 2023-09-30T01:20:10.714 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1924: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2023-09-30T01:20:10.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: test_expect_failure: teardown td/ceph-helpers 2023-09-30T01:20:10.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:20:10.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:20:10.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:20:10.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:10.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:10.717 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:10.717 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:10.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:10.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:10.721 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:20:10.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:20:10.723 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:20:10.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:20:10.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:20:10.726 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:20:10.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:10.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:20:10.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:20:10.729 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:10.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:20:10.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:20:10.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:20:10.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:20:10.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:10.735 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:10.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:20:10.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:20:10.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:20:10.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:20:10.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_flush_pg_stats td/ceph-helpers 2023-09-30T01:20:10.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: test_flush_pg_stats: local dir=td/ceph-helpers 2023-09-30T01:20:10.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: test_flush_pg_stats: setup td/ceph-helpers 2023-09-30T01:20:10.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:20:10.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:20:10.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:20:10.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:20:10.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:20:10.739 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:10.739 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:10.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:10.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:10.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:10.742 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:10.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:20:10.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:20:10.745 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:20:10.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:20:10.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:20:10.747 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:20:10.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:10.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:20:10.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:20:10.749 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:10.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:20:10.752 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:20:10.752 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:20:10.754 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:20:10.754 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:10.755 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:10.755 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:20:10.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:20:10.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:20:10.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:20:10.759 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:20:10.759 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:10.759 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:10.759 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:20:10.762 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:20:10.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:20:10.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:20:10.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:20:10.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:20:10.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:20:10.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:20:10.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:20:10.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:20:10.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:20:10.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:20:11.139 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:20:11.139 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:20:11.139 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:20:11.140 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:20:11.140 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:11.140 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:11.140 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:20:11.141 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:20:11.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:20:11.454 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:20:11.454 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:20:11.454 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:20:11.454 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:20:11.455 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:20:11.455 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:20:11.455 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:20:11.455 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:20:11.455 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:20:11.456 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:11.456 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:11.456 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:20:11.456 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:20:11.456 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:20:11.547 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:20:11.547 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:20:11.547 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:20:11.547 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:20:11.548 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:20:11.548 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:20:11.549 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:20:11.549 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:20:11.549 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:20:11.549 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:11.549 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:11.549 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:20:11.550 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:20:11.550 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:20:11.633 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_flush_pg_stats: run_mgr td/ceph-helpers x 2023-09-30T01:20:11.633 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:20:11.633 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:20:11.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:20:11.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:20:11.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:20:11.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:20:12.542 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:20:12.542 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:20:12.542 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:20:12.543 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:20:12.543 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:12.543 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:12.543 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:20:12.545 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:20:12.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:20:12.580 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: test_flush_pg_stats: run_osd td/ceph-helpers 0 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:20:12.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:20:12.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:20:12.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:20:12.583 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:20:12.583 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:20:12.583 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:20:12.583 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:20:12.583 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:12.583 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:20:12.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:20:12.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:20:12.585 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:20:12.586 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1f4f697c-b3ae-4f94-ac27-aa1f28b3454e 2023-09-30T01:20:12.586 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 1f4f697c-b3ae-4f94-ac27-aa1f28b3454e 2023-09-30T01:20:12.587 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1f4f697c-b3ae-4f94-ac27-aa1f28b3454e' 2023-09-30T01:20:12.587 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:20:12.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMdxdl3I61JBAAgywDb3CDuc2ZRPHw9EdXIg== 2023-09-30T01:20:12.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMdxdl3I61JBAAgywDb3CDuc2ZRPHw9EdXIg=="}' 2023-09-30T01:20:12.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1f4f697c-b3ae-4f94-ac27-aa1f28b3454e -i td/ceph-helpers/0/new.json 2023-09-30T01:20:12.979 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:20:12.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:20:12.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMdxdl3I61JBAAgywDb3CDuc2ZRPHw9EdXIg== --osd-uuid 1f4f697c-b3ae-4f94-ac27-aa1f28b3454e 2023-09-30T01:20:13.026 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:13.021+0000 7fb818ea93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:20:13.029 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:13.025+0000 7fb818ea93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:20:13.031 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:13.025+0000 7fb818ea93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:20:13.031 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:13.025+0000 7fb818ea93c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:20:13.031 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:13.025+0000 7fb818ea93c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:20:13.031 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:13.025+0000 7fb818ea93c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:20:13.111 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:13.105+0000 7fb818ea93c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:20:19.466 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:20:19.466 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:20:19.468 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:20:19.468 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:20:19.469 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:20:20.044 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:20:20.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:20:20.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:20:20.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:20:20.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:20:20.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:20:20.073 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:20.069+0000 7f02286083c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:20:20.086 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:20.081+0000 7f02286083c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:20:20.087 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:20.081+0000 7f02286083c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:20:20.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:20:20.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:20:20.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:20:20.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:20:20.464 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:20:20.464 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:20.464 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:20:20.464 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:20:20.465 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:20.465 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:20.635 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:20.629+0000 7f02286083c0 -1 Falling back to public interface 2023-09-30T01:20:20.878 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:21.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:21.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:21.880 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:20:21.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:20:21.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:21.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:22.271 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:23.273 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:23.273 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:23.274 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:20:23.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:20:23.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:23.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:23.286 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:23.281+0000 7f02286083c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:20:23.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:24.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:24.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:24.724 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:20:24.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:20:24.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:24.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:25.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:26.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:26.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:26.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:20:26.154 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:20:26.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:26.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:26.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:27.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:27.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:27.557 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:20:27.558 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:20:27.558 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:27.558 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:28.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:29.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:29.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:29.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:20:29.005 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:20:29.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:29.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:29.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:30.417 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:30.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:30.418 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:20:30.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:20:30.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:30.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:30.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:31.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:31.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:31.814 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:20:31.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:20:31.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:31.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:32.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:33.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:33.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:33.361 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:20:33.361 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:20:33.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:33.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:33.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:34.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:34.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:34.758 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:20:34.759 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:20:34.760 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:34.760 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:35.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:36.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:36.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:36.648 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:20:36.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:20:36.649 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:36.649 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:37.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:38.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:38.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:38.162 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:20:38.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:20:38.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:38.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:39.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:40.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:40.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:40.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:20:40.008 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:20:40.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:40.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:40.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:41.714 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:41.714 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:41.714 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:20:41.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:20:41.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:41.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:42.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:43.361 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:43.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:43.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:20:43.362 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:20:43.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:43.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:43.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:20:44.943 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:20:44.943 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:20:44.944 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:20:44.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:20:44.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:20:44.945 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:20:45.450 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/88168,v1:127.0.0.1:6803/88168] [v2:127.0.0.1:6804/88168,v1:127.0.0.1:6805/88168] exists,up 1f4f697c-b3ae-4f94-ac27-aa1f28b3454e 2023-09-30T01:20:45.450 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:20:45.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:20:45.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:20:45.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: test_flush_pg_stats: create_rbd_pool 2023-09-30T01:20:45.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:20:45.893 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:20:45.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:20:45.903 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:20:46.521 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:20:46.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:20:47.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:20:48.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2164: test_flush_pg_stats: rados -p rbd put obj /etc/group 2023-09-30T01:20:48.610 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: test_flush_pg_stats: flush_pg_stats 2023-09-30T01:20:48.610 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:20:48.611 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:20:49.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:20:49.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:20:49.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:20:49.118 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:20:49.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836490 2023-09-30T01:20:49.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836490 2023-09-30T01:20:49.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836490' 2023-09-30T01:20:49.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:20:49.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836490 2023-09-30T01:20:49.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:20:49.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:20:49.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836490 2023-09-30T01:20:49.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:20:49.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836490 2023-09-30T01:20:49.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2023-09-30T01:20:49.337 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836490 2023-09-30T01:20:49.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:20:49.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836488 -lt 21474836490 2023-09-30T01:20:49.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:20:50.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:20:50.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:20:51.089 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836490 -lt 21474836490 2023-09-30T01:20:51.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2023-09-30T01:20:51.091 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: test_flush_pg_stats: ceph df detail --format=json 2023-09-30T01:20:51.091 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2023-09-30T01:20:51.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: test_flush_pg_stats: stored=5985 2023-09-30T01:20:51.662 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: test_flush_pg_stats: ceph df detail --format=json 2023-09-30T01:20:51.663 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2023-09-30T01:20:52.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: test_flush_pg_stats: stored_raw=5985 2023-09-30T01:20:52.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: test_flush_pg_stats: test 5985 -gt 0 2023-09-30T01:20:52.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: test_flush_pg_stats: test 5985 == 5985 2023-09-30T01:20:52.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: test_flush_pg_stats: teardown td/ceph-helpers 2023-09-30T01:20:52.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:20:52.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:20:52.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:20:52.232 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:52.232 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:52.232 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:52.232 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:52.232 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:52.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:52.345 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:20:52.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:20:52.347 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:20:52.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:20:52.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:20:52.350 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:20:52.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:52.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:20:52.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:20:52.353 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:52.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:20:52.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:20:52.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:20:52.366 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:20:52.366 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:52.366 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:52.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:20:52.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:20:52.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:20:52.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:20:52.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_config td/ceph-helpers 2023-09-30T01:20:52.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: local dir=td/ceph-helpers 2023-09-30T01:20:52.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: test_get_config: setup td/ceph-helpers 2023-09-30T01:20:52.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:20:52.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:20:52.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:20:52.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:20:52.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:20:52.370 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:20:52.370 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:20:52.371 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:20:52.371 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:20:52.371 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:20:52.373 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:20:52.374 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:20:52.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:20:52.376 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:20:52.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:20:52.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:20:52.378 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:20:52.379 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:52.379 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:20:52.380 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:20:52.380 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:20:52.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:20:52.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:20:52.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:20:52.386 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:20:52.386 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:52.386 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:52.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:20:52.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:20:52.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:20:52.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:20:52.390 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:20:52.390 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:52.390 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:52.390 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:20:52.392 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:20:52.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:20:52.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:20:52.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:20:52.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1140: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:20:52.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:20:52.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:20:52.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:20:52.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:20:52.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:20:52.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:20:52.808 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:20:52.809 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:20:52.809 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:20:52.809 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:20:52.809 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:52.809 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:52.810 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:20:52.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:20:53.089 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:20:53.091 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:20:53.092 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:20:53.092 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:20:53.092 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:20:53.093 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:20:53.093 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:20:53.093 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:20:53.094 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:20:53.095 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:20:53.095 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:53.095 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:53.095 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:20:53.096 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:20:53.096 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:20:53.185 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:20:53.185 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:20:53.186 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:20:53.186 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:20:53.186 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:20:53.186 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:20:53.187 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:20:53.187 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:20:53.187 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:20:53.187 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:53.187 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:53.188 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:20:53.188 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:20:53.188 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:20:53.273 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: get_config mon a osd_pool_default_size 2023-09-30T01:20:53.273 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:20:53.273 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:20:53.274 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_pool_default_size 2023-09-30T01:20:53.274 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_pool_default_size 2023-09-30T01:20:53.276 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:20:53.276 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:20:53.276 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:20:53.276 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:20:53.276 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:53.276 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:53.277 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:20:53.277 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:20:53.277 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get osd_pool_default_size 2023-09-30T01:20:53.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: test 1 = 1 2023-09-30T01:20:53.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: test_get_config: run_mgr td/ceph-helpers x 2023-09-30T01:20:53.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:20:53.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:20:53.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:20:53.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:20:53.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:20:53.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:20:54.258 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:20:54.258 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:20:54.258 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:20:54.259 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:20:54.259 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:54.259 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:54.259 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:20:54.260 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:20:54.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:20:54.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1143: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2023-09-30T01:20:54.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:20:54.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:20:54.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:20:54.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:20:54.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:20:54.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:20:54.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:20:54.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:20:54.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:20:54.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:20:54.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:20:54.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:20:54.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:20:54.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:20:54.328 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:20:54.328 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:20:54.328 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:20:54.329 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:20:54.329 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:20:54.330 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:20:54.330 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:20:54.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:20:54.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:20:54.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:20:54.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:20:54.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:20:54.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:20:54.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:20:54.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:20:54.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:20:54.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:20:54.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:20:54.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_max_scrubs=3 2023-09-30T01:20:54.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:20:54.331 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:20:54.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5eca1cc8-0895-4409-b69d-44ef0dd96a22 2023-09-30T01:20:54.332 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 5eca1cc8-0895-4409-b69d-44ef0dd96a22 2023-09-30T01:20:54.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5eca1cc8-0895-4409-b69d-44ef0dd96a22' 2023-09-30T01:20:54.333 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:20:54.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2dxdlZvpyFRAANUB0tLOW8BGDhlvM6nMe/A== 2023-09-30T01:20:54.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2dxdlZvpyFRAANUB0tLOW8BGDhlvM6nMe/A=="}' 2023-09-30T01:20:54.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5eca1cc8-0895-4409-b69d-44ef0dd96a22 -i td/ceph-helpers/0/new.json 2023-09-30T01:20:54.757 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:20:54.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:20:54.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQD2dxdlZvpyFRAANUB0tLOW8BGDhlvM6nMe/A== --osd-uuid 5eca1cc8-0895-4409-b69d-44ef0dd96a22 2023-09-30T01:20:54.804 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:54.797+0000 7ffaedc4a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:20:54.807 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:54.801+0000 7ffaedc4a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:20:54.808 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:54.805+0000 7ffaedc4a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:20:54.809 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:54.805+0000 7ffaedc4a3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:20:54.809 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:54.805+0000 7ffaedc4a3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:20:54.809 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:54.805+0000 7ffaedc4a3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:20:54.889 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:20:54.885+0000 7ffaedc4a3c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:21:01.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:21:01.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:21:01.136 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:21:01.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:21:01.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:21:01.712 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:21:01.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:21:01.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:21:01.714 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2023-09-30T01:21:01.714 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:21:01.714 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:21:01.746 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:01.740+0000 7f87c65f13c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:21:01.749 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:01.744+0000 7f87c65f13c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:21:01.753 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:01.744+0000 7f87c65f13c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:21:02.043 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:02.036+0000 7f87c65f13c0 -1 Falling back to public interface 2023-09-30T01:21:02.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:21:02.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:21:02.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:21:02.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:21:02.095 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:21:02.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:21:02.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:02.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:21:02.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:02.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:02.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:03.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:03.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:03.486 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:21:03.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:21:03.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:03.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:03.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:04.607 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:04.600+0000 7f87c65f13c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:21:04.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:04.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:04.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:21:04.893 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:21:04.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:04.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:05.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:06.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:06.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:06.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:21:06.384 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:21:06.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:06.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:06.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:07.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:07.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:07.814 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:21:07.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:21:07.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:07.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:08.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:09.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:09.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:09.335 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:21:09.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:21:09.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:09.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:09.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:10.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:10.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:10.772 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:21:10.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:21:10.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:10.773 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:11.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:12.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:12.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:12.301 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:21:12.301 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:21:12.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:12.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:12.700 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:13.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:13.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:13.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:21:13.702 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:21:13.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:13.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:14.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:15.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:15.117 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:15.117 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:21:15.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:21:15.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:15.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:15.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:16.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:16.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:16.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:21:16.961 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:21:16.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:16.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:18.103 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:19.105 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:19.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:19.106 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:21:19.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:21:19.107 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:19.107 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:20.023 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:21.024 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:21.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:21.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:21:21.025 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:21:21.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:21.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:22.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:23.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:23.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:23.115 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:21:23.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:21:23.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:23.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:23.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:24.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:24.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:24.959 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:21:24.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:21:24.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:24.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:25.904 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:26.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:26.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:26.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:21:26.907 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:21:26.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:26.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:27.313 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/90460,v1:127.0.0.1:6803/90460] [v2:127.0.0.1:6804/90460,v1:127.0.0.1:6805/90460] exists,up 5eca1cc8-0895-4409-b69d-44ef0dd96a22 2023-09-30T01:21:27.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:21:27.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:21:27.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:21:27.315 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: get_config osd 0 osd_max_scrubs 2023-09-30T01:21:27.315 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2023-09-30T01:21:27.315 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2023-09-30T01:21:27.315 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_max_scrubs 2023-09-30T01:21:27.316 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_max_scrubs 2023-09-30T01:21:27.316 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2023-09-30T01:21:27.316 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:21:27.316 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:21:27.317 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:21:27.317 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:27.317 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:27.317 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.0.asok 2023-09-30T01:21:27.318 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:21:27.318 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-osd.0.asok config get osd_max_scrubs 2023-09-30T01:21:27.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: test 3 = 3 2023-09-30T01:21:27.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1145: test_get_config: teardown td/ceph-helpers 2023-09-30T01:21:27.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:21:27.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:21:27.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:21:27.410 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:21:27.410 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:21:27.410 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:21:27.410 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:21:27.410 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:21:27.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:21:27.521 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:21:27.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:21:27.523 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:21:27.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:21:27.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:21:27.526 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:21:27.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:21:27.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:21:27.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:21:27.528 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:21:27.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:21:27.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:21:27.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:21:27.542 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:21:27.542 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:27.542 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:27.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:21:27.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:21:27.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:21:27.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:21:27.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2023-09-30T01:21:27.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2023-09-30T01:21:27.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: test_get_is_making_recovery_progress: setup td/ceph-helpers 2023-09-30T01:21:27.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:21:27.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:21:27.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:21:27.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:21:27.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:21:27.546 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:21:27.546 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:21:27.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:21:27.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:21:27.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:21:27.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:21:27.550 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:21:27.551 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:21:27.552 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:21:27.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:21:27.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:21:27.555 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:21:27.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:21:27.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:21:27.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:21:27.557 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:21:27.558 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:21:27.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:21:27.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:21:27.562 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:21:27.562 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:27.562 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:27.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:21:27.564 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:21:27.564 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:21:27.564 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:21:27.566 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:21:27.566 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:27.566 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:27.567 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:21:27.569 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:21:27.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:21:27.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:21:27.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:21:27.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1347: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2023-09-30T01:21:27.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:21:27.571 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:21:27.571 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:21:27.571 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:21:27.571 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:21:27.571 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2023-09-30T01:21:27.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:21:27.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:21:27.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:21:27.933 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:21:27.934 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:27.934 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:27.934 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:21:27.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:21:28.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:21:28.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:21:28.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:21:28.252 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:21:28.252 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:21:28.252 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:21:28.252 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:21:28.252 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:21:28.253 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:21:28.254 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:21:28.254 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:28.254 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:28.254 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:21:28.254 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:21:28.255 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:21:28.350 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:21:28.350 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:21:28.350 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:21:28.350 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:21:28.351 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:21:28.351 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:21:28.351 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:21:28.351 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:21:28.351 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:21:28.352 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:28.352 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:28.352 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:21:28.352 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:21:28.352 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:21:28.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1348: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2023-09-30T01:21:28.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:21:28.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:21:28.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:21:28.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:21:28.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:21:28.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:21:29.339 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:21:29.339 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:21:29.339 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:21:29.340 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:21:29.340 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:29.340 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:29.340 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:21:29.341 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:21:29.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:21:29.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1349: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2023-09-30T01:21:29.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-09-30T01:21:29.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-09-30T01:21:29.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-09-30T01:21:29.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-09-30T01:21:29.379 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-09-30T01:21:29.379 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-09-30T01:21:29.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-09-30T01:21:29.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-09-30T01:21:29.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1350: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2023-09-30T01:21:29.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:21:29.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:21:29.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:21:29.652 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:21:29.652 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:21:29.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:21:29.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:21:29.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:21:29.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:21:29.764 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:21:29.764 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:21:29.765 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:21:29.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:21:29.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:21:29.768 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:21:29.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:21:29.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:21:29.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:21:29.771 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:21:29.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:21:29.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:21:29.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:21:29.778 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:21:29.778 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:29.778 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:29.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:21:29.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:21:29.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:21:29.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:21:29.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2023-09-30T01:21:29.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2023-09-30T01:21:29.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1535: test_get_last_scrub_stamp: setup td/ceph-helpers 2023-09-30T01:21:29.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:21:29.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:21:29.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:21:29.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:21:29.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:21:29.783 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:21:29.783 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:21:29.783 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:21:29.784 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:21:29.784 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:21:29.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:21:29.786 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:21:29.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:21:29.788 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:21:29.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:21:29.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:21:29.791 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:21:29.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:21:29.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:21:29.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:21:29.793 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:21:29.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:21:29.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:21:29.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:21:29.799 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:21:29.799 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:29.799 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:29.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:21:29.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:21:29.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:21:29.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:21:29.803 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:21:29.804 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:29.804 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:29.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:21:29.806 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:21:29.807 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:21:29.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:21:29.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:21:29.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1536: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:21:29.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:21:29.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:21:29.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:21:29.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:21:29.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:21:29.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:21:30.232 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:21:30.232 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:21:30.232 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:21:30.233 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:21:30.233 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:30.233 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:30.233 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:21:30.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:21:30.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:21:30.515 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:21:30.515 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:21:30.515 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:21:30.515 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:21:30.515 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:21:30.516 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:21:30.516 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:21:30.516 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:21:30.517 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:21:30.517 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:30.517 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:30.517 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:21:30.517 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:21:30.518 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:21:30.609 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:21:30.610 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:21:30.610 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:21:30.610 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:21:30.611 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:21:30.611 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:21:30.611 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:21:30.611 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:21:30.612 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:21:30.612 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:30.612 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:30.612 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:21:30.612 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:21:30.612 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:21:30.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1537: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2023-09-30T01:21:30.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:21:30.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:21:30.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:21:30.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:21:30.692 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:21:30.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:21:31.587 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:21:31.588 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:21:31.588 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:21:31.588 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:21:31.588 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:31.589 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:31.589 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:21:31.589 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:21:31.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:21:31.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1538: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2023-09-30T01:21:31.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:21:31.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:21:31.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:21:31.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:21:31.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:21:31.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:21:31.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:21:31.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:21:31.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:21:31.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:21:31.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:21:31.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:21:31.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:21:31.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:21:31.628 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:21:31.628 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:21:31.628 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:21:31.630 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:21:31.630 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:21:31.630 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:21:31.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:21:31.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:21:31.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:21:31.632 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:21:31.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e944a198-2e00-4ee6-b662-f25253bee60e 2023-09-30T01:21:31.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e944a198-2e00-4ee6-b662-f25253bee60e' 2023-09-30T01:21:31.633 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 e944a198-2e00-4ee6-b662-f25253bee60e 2023-09-30T01:21:31.633 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:21:31.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbeBdlpmW9JhAAA8qzyRd0YNf53nyTDIpfaQ== 2023-09-30T01:21:31.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbeBdlpmW9JhAAA8qzyRd0YNf53nyTDIpfaQ=="}' 2023-09-30T01:21:31.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e944a198-2e00-4ee6-b662-f25253bee60e -i td/ceph-helpers/0/new.json 2023-09-30T01:21:32.031 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:21:32.044 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:21:32.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbeBdlpmW9JhAAA8qzyRd0YNf53nyTDIpfaQ== --osd-uuid e944a198-2e00-4ee6-b662-f25253bee60e 2023-09-30T01:21:32.071 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:32.064+0000 7fddecaa93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:21:32.079 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:32.072+0000 7fddecaa93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:21:32.081 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:32.076+0000 7fddecaa93c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:21:32.082 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:32.076+0000 7fddecaa93c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:21:32.082 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:32.076+0000 7fddecaa93c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:21:32.082 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:32.076+0000 7fddecaa93c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:21:32.180 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:32.172+0000 7fddecaa93c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:21:38.474 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:21:38.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:21:38.476 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:21:38.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:21:38.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:21:39.071 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:21:39.071 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:21:39.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:21:39.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:21:39.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:21:39.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:21:39.100 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:39.096+0000 7f21d242e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:21:39.105 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:39.100+0000 7f21d242e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:21:39.109 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:39.104+0000 7f21d242e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:21:39.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:21:39.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:21:39.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:21:39.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:21:39.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:21:39.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:39.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:21:39.481 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:21:39.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:39.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:39.671 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:39.664+0000 7f21d242e3c0 -1 Falling back to public interface 2023-09-30T01:21:39.906 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:40.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:40.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:40.908 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:21:40.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:21:40.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:40.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:41.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:42.174 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:21:42.168+0000 7f21d242e3c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:21:42.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:42.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:42.328 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:21:42.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:21:42.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:42.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:42.856 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:43.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:43.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:43.858 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:21:43.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:21:43.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:43.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:44.263 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:45.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:45.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:45.265 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:21:45.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:21:45.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:45.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:45.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:46.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:46.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:46.664 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:21:46.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:21:46.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:46.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:47.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:48.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:48.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:48.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:21:48.192 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:21:48.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:48.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:48.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:49.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:49.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:49.599 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:21:49.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:21:49.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:49.600 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:50.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:51.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:51.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:51.137 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:21:51.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:21:51.138 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:51.138 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:51.551 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:52.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:52.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:52.553 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:21:52.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:21:52.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:52.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:53.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:54.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:54.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:54.130 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:21:54.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:21:54.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:54.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:54.676 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:55.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:55.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:55.678 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:21:55.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:21:55.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:55.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:56.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:57.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:57.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:57.334 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:21:57.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:21:57.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:57.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:21:58.398 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:21:59.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:21:59.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:21:59.401 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:21:59.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:21:59.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:21:59.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:00.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:01.071 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:01.071 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:01.071 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:22:01.072 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:22:01.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:01.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:01.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:02.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:02.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:02.758 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:22:02.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:22:02.759 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:02.759 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:03.575 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:04.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:04.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:04.577 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:22:04.578 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:22:04.578 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:04.578 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:04.988 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/92687,v1:127.0.0.1:6803/92687] [v2:127.0.0.1:6804/92687,v1:127.0.0.1:6805/92687] exists,up e944a198-2e00-4ee6-b662-f25253bee60e 2023-09-30T01:22:04.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:22:04.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:22:04.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:22:04.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1539: test_get_last_scrub_stamp: create_rbd_pool 2023-09-30T01:22:04.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:22:05.411 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:22:05.418 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:22:05.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:22:06.122 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:22:06.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:22:07.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:22:08.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1540: test_get_last_scrub_stamp: wait_for_clean 2023-09-30T01:22:08.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:22:08.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:22:08.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:22:08.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:22:08.085 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:22:08.086 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:22:08.086 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:22:08.087 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:22:08.087 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:22:08.087 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:22:08.295 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:22:08.295 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:22:08.295 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:22:08.295 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:22:08.296 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:22:08.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:22:08.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:22:08.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:22:08.746 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:22:08.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836489 2023-09-30T01:22:08.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836489 2023-09-30T01:22:08.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836489' 2023-09-30T01:22:08.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:22:08.894 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836489 2023-09-30T01:22:08.894 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:22:08.896 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:22:08.897 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836489 2023-09-30T01:22:08.898 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:22:08.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836489 2023-09-30T01:22:08.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2023-09-30T01:22:08.900 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836489 2023-09-30T01:22:08.900 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:22:09.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836487 -lt 21474836489 2023-09-30T01:22:09.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:22:10.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:22:10.339 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:22:10.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836487 -lt 21474836489 2023-09-30T01:22:10.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:22:11.759 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 299 -eq 0 ']' 2023-09-30T01:22:11.759 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:22:12.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836490 -lt 21474836489 2023-09-30T01:22:12.164 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:22:12.165 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:22:12.165 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:22:12.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:22:12.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:22:12.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:22:12.737 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:22:12.737 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:22:12.737 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:22:12.737 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:22:12.737 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:22:13.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:22:13.145 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:22:13.145 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:22:13.146 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:22:13.630 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:22:13.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:22:13.631 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:22:13.631 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2023-09-30T01:22:13.631 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:22:13.632 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:22:13.632 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:22:13.632 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:22:14.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: stamp=2023-09-30T01:22:05.793286+0000 2023-09-30T01:22:14.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1542: test_get_last_scrub_stamp: test -n 2023-09-30T01:22:05.793286+0000 2023-09-30T01:22:14.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1543: test_get_last_scrub_stamp: teardown td/ceph-helpers 2023-09-30T01:22:14.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:22:14.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:22:14.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:22:14.064 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:22:14.064 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:22:14.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:22:14.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:22:14.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:22:14.181 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:22:14.182 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:22:14.182 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:22:14.182 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:22:14.183 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:22:14.183 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:22:14.184 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:22:14.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:22:14.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:22:14.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:22:14.187 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:22:14.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:22:14.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:22:14.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:22:14.200 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:22:14.200 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:22:14.200 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:22:14.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:22:14.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:22:14.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:22:14.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:22:14.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_not_primary td/ceph-helpers 2023-09-30T01:22:14.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local dir=td/ceph-helpers 2023-09-30T01:22:14.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: setup td/ceph-helpers 2023-09-30T01:22:14.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:22:14.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:22:14.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:22:14.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:22:14.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:22:14.207 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:22:14.207 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:22:14.207 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:22:14.207 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:22:14.207 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:22:14.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:22:14.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:22:14.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:22:14.211 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:22:14.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:22:14.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:22:14.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:22:14.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:22:14.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:22:14.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:22:14.217 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:22:14.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:22:14.220 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:22:14.220 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:22:14.222 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:22:14.222 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:22:14.222 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:22:14.223 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:22:14.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:22:14.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:22:14.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:22:14.226 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:22:14.227 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:22:14.227 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:22:14.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:22:14.229 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:22:14.230 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:22:14.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:22:14.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:22:14.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2023-09-30T01:22:14.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:22:14.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:22:14.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:22:14.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:22:14.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:22:14.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2023-09-30T01:22:14.598 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:22:14.599 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:22:14.599 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:22:14.599 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:22:14.599 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:22:14.599 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:22:14.600 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:22:14.600 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2023-09-30T01:22:14.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:22:14.881 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:22:14.881 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:22:14.881 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:22:14.881 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:22:14.881 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:22:14.882 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:22:14.882 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:22:14.882 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:22:14.883 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:22:14.883 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:22:14.883 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:22:14.883 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:22:14.884 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:22:14.884 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:22:14.986 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:22:14.987 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:22:14.987 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:22:14.987 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:22:14.988 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:22:14.988 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:22:14.988 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:22:14.989 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:22:14.989 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:22:14.989 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:22:14.989 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:22:14.989 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:22:14.989 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:22:14.990 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:22:15.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: run_mgr td/ceph-helpers x 2023-09-30T01:22:15.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:22:15.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:22:15.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:22:15.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:22:15.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:22:15.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:22:15.988 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:22:15.988 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:22:15.988 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:22:15.989 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:22:15.989 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:22:15.989 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:22:15.989 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:22:15.990 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:22:15.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:22:16.057 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1243: test_get_not_primary: run_osd td/ceph-helpers 0 2023-09-30T01:22:16.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:22:16.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:22:16.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:22:16.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:22:16.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:22:16.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:22:16.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:22:16.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:22:16.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:22:16.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:22:16.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:22:16.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:22:16.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:22:16.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:22:16.061 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:22:16.061 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:22:16.061 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:22:16.061 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:22:16.061 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:22:16.062 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:22:16.062 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:22:16.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:22:16.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:22:16.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:22:16.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:22:16.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:22:16.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:22:16.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:22:16.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:22:16.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:22:16.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:22:16.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:22:16.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:22:16.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:22:16.063 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:22:16.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c70eafc3-be4a-412b-b8b3-df730eb7c7f4 2023-09-30T01:22:16.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c70eafc3-be4a-412b-b8b3-df730eb7c7f4' 2023-09-30T01:22:16.065 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 c70eafc3-be4a-412b-b8b3-df730eb7c7f4 2023-09-30T01:22:16.065 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:22:16.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIeBdlH0kUBRAAaJgMWsGrtEOhJcLDWHTw4Q== 2023-09-30T01:22:16.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIeBdlH0kUBRAAaJgMWsGrtEOhJcLDWHTw4Q=="}' 2023-09-30T01:22:16.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c70eafc3-be4a-412b-b8b3-df730eb7c7f4 -i td/ceph-helpers/0/new.json 2023-09-30T01:22:16.440 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:22:16.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:22:16.454 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIeBdlH0kUBRAAaJgMWsGrtEOhJcLDWHTw4Q== --osd-uuid c70eafc3-be4a-412b-b8b3-df730eb7c7f4 2023-09-30T01:22:16.482 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:16.479+0000 7f642b8e13c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:16.484 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:16.483+0000 7f642b8e13c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:16.486 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:16.483+0000 7f642b8e13c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:16.488 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:16.483+0000 7f642b8e13c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:22:16.488 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:16.483+0000 7f642b8e13c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:22:16.488 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:16.483+0000 7f642b8e13c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:22:16.571 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:16.567+0000 7f642b8e13c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:22:22.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:22:22.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:22:22.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:22:22.933 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:22:22.933 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:22:23.471 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:22:23.471 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:22:23.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:22:23.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:22:23.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:22:23.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:22:23.502 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:23.499+0000 7f35fbbd83c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:23.529 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:23.527+0000 7f35fbbd83c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:23.530 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:23.527+0000 7f35fbbd83c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:23.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:22:23.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:22:23.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:22:23.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:22:23.964 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:22:23.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:22:23.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:23.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:22:23.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:23.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:24.087 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:24.083+0000 7f35fbbd83c0 -1 Falling back to public interface 2023-09-30T01:22:24.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:25.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:25.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:25.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:22:25.446 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:22:25.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:25.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:25.835 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:26.638 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:26.635+0000 7f35fbbd83c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:22:26.837 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:26.837 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:26.837 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:22:26.837 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:22:26.838 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:26.838 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:27.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:28.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:28.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:28.333 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:22:28.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:22:28.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:28.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:28.742 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:29.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:29.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:29.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:22:29.746 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:22:29.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:29.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:30.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:31.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:31.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:31.217 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:22:31.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:22:31.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:31.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:31.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:32.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:32.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:32.685 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:22:32.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:22:32.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:32.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:33.086 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:34.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:34.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:34.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:22:34.089 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:22:34.089 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:34.089 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:34.500 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:35.501 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:35.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:35.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:22:35.502 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:22:35.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:35.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:35.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:36.925 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:36.925 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:36.926 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:22:36.926 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:22:36.926 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:36.926 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:37.442 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:38.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:38.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:38.445 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:22:38.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:22:38.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:38.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:39.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:40.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:40.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:40.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:22:40.242 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:22:40.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:40.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:40.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:41.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:41.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:41.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:22:41.814 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:22:41.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:41.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:42.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:43.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:43.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:43.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:22:43.552 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:22:43.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:43.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:44.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:45.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:45.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:45.548 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:22:45.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:22:45.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:45.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:46.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:47.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:47.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:47.214 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:22:47.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:22:47.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:47.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:48.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:49.020 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:49.021 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:49.021 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:22:49.021 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:22:49.021 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:49.021 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:22:49.488 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/95136,v1:127.0.0.1:6803/95136] [v2:127.0.0.1:6804/95136,v1:127.0.0.1:6805/95136] exists,up c70eafc3-be4a-412b-b8b3-df730eb7c7f4 2023-09-30T01:22:49.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:22:49.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:22:49.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:22:49.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: test_get_not_primary: run_osd td/ceph-helpers 1 2023-09-30T01:22:49.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:22:49.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:22:49.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-09-30T01:22:49.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:22:49.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2023-09-30T01:22:49.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:22:49.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:22:49.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:22:49.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:22:49.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2023-09-30T01:22:49.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2023-09-30T01:22:49.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:22:49.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:22:49.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:22:49.492 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:22:49.492 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:22:49.492 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:22:49.492 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:22:49.492 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:22:49.492 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:22:49.493 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:22:49.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:22:49.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:22:49.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:22:49.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:22:49.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:22:49.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:22:49.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:22:49.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:22:49.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:22:49.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:22:49.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:22:49.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:22:49.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2023-09-30T01:22:49.495 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:22:49.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a88c1922-e691-4c98-b395-bd56b1f4510b 2023-09-30T01:22:49.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a88c1922-e691-4c98-b395-bd56b1f4510b' 2023-09-30T01:22:49.496 INFO:tasks.workunit.client.0.smithi186.stdout:add osd1 a88c1922-e691-4c98-b395-bd56b1f4510b 2023-09-30T01:22:49.496 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:22:49.516 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpeBdlw+CyHhAAXrhoTp1mR8IqlUQZz6Tifw== 2023-09-30T01:22:49.516 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpeBdlw+CyHhAAXrhoTp1mR8IqlUQZz6Tifw=="}' 2023-09-30T01:22:49.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a88c1922-e691-4c98-b395-bd56b1f4510b -i td/ceph-helpers/1/new.json 2023-09-30T01:22:50.025 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:22:50.038 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2023-09-30T01:22:50.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpeBdlw+CyHhAAXrhoTp1mR8IqlUQZz6Tifw== --osd-uuid a88c1922-e691-4c98-b395-bd56b1f4510b 2023-09-30T01:22:50.067 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:50.063+0000 7fcd860de3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:50.070 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:50.067+0000 7fcd860de3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:50.072 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:50.067+0000 7fcd860de3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:50.072 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:50.071+0000 7fcd860de3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:22:50.072 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:50.071+0000 7fcd860de3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:22:50.072 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:50.071+0000 7fcd860de3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:22:50.147 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:50.143+0000 7fcd860de3c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2023-09-30T01:22:56.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2023-09-30T01:22:56.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:22:56.759 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd1 key to auth repository 2023-09-30T01:22:56.760 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-09-30T01:22:56.760 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:22:57.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-09-30T01:22:57.378 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:22:57.379 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:22:57.379 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:22:57.379 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:22:57.379 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:22:57.408 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:57.403+0000 7ffabcf443c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:57.432 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:57.431+0000 7ffabcf443c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:57.434 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:57.431+0000 7ffabcf443c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:22:57.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-09-30T01:22:57.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:22:57.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:22:57.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:22:57.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:22:57.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:57.758 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:22:57.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:22:57.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:57.759 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:22:57.970 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:22:57.967+0000 7ffabcf443c0 -1 Falling back to public interface 2023-09-30T01:22:58.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:22:59.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:22:59.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:22:59.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:22:59.187 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:22:59.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:22:59.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:22:59.574 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:00.540 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:00.538+0000 7ffabcf443c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:23:00.576 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:00.576 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:00.576 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:23:00.577 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:23:00.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:00.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:01.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:02.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:02.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:02.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:23:02.200 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:23:02.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:02.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:02.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:03.637 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:03.637 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:03.637 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:23:03.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:23:03.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:03.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:04.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:05.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:05.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:05.092 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:23:05.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:23:05.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:05.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:05.773 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:06.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:06.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:06.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:23:06.777 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:23:06.777 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:06.777 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:07.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:08.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:08.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:08.340 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:23:08.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:23:08.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:08.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:08.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:09.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:09.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:09.772 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:23:09.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:23:09.773 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:09.773 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:10.344 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:11.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:11.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:11.346 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:23:11.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:23:11.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:11.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:11.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:12.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:12.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:12.974 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:23:12.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:23:12.975 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:12.975 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:14.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:15.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:15.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:15.011 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:23:15.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:23:15.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:15.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:15.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:16.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:16.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:16.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:23:16.747 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:23:16.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:16.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:17.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:18.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:18.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:18.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:23:18.804 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:23:18.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:18.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:20.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:21.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:21.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:21.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:23:21.065 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:23:21.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:21.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:21.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:22.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:22.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:22.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:23:22.852 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:23:22.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:22.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:23:23.261 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/96886,v1:127.0.0.1:6811/96886] [v2:127.0.0.1:6812/96886,v1:127.0.0.1:6813/96886] exists,up a88c1922-e691-4c98-b395-bd56b1f4510b 2023-09-30T01:23:23.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:23:23.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:23:23.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:23:23.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: test_get_not_primary: create_rbd_pool 2023-09-30T01:23:23.263 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:23:23.718 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:23:23.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:23:23.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:23:24.415 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:23:24.425 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:23:25.427 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:23:26.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1246: test_get_not_primary: wait_for_clean 2023-09-30T01:23:26.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:23:26.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:23:26.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:23:26.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:23:26.990 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:23:26.991 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:23:26.991 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:23:26.992 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:23:26.992 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:23:26.992 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:23:27.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:23:27.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:23:27.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:23:27.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:23:27.200 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:23:27.605 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2023-09-30T01:23:27.605 INFO:tasks.workunit.client.0.smithi186.stderr:1' 2023-09-30T01:23:27.605 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:23:27.605 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:23:27.606 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:23:27.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836498 2023-09-30T01:23:27.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836498 2023-09-30T01:23:27.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836498' 2023-09-30T01:23:27.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:23:27.790 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-09-30T01:23:27.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672970 2023-09-30T01:23:27.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672970 2023-09-30T01:23:27.936 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836498 1-42949672970' 2023-09-30T01:23:27.936 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:23:27.936 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836498 2023-09-30T01:23:27.936 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:23:27.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:23:27.938 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836498 2023-09-30T01:23:27.939 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:23:27.940 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836498 2023-09-30T01:23:27.940 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2023-09-30T01:23:27.941 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836498 2023-09-30T01:23:27.941 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:23:28.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836496 -lt 21474836498 2023-09-30T01:23:28.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:23:29.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:23:29.334 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:23:29.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836498 -lt 21474836498 2023-09-30T01:23:29.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:23:29.729 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:23:29.729 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672970 2023-09-30T01:23:29.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2023-09-30T01:23:29.732 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672970 2023-09-30T01:23:29.732 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:23:29.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672970 2023-09-30T01:23:29.734 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.1 seq 42949672970 2023-09-30T01:23:29.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2023-09-30T01:23:29.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:23:30.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672971 -lt 42949672970 2023-09-30T01:23:30.163 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:23:30.163 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:23:30.163 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:23:30.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:23:30.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:23:30.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:23:30.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:23:30.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:23:30.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:23:30.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:23:30.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:23:31.111 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:23:31.111 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:23:31.111 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:23:31.112 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:23:31.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:23:31.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:23:31.717 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:23:31.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: get_primary rbd GROUP 2023-09-30T01:23:31.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2023-09-30T01:23:31.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2023-09-30T01:23:31.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2023-09-30T01:23:31.718 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2023-09-30T01:23:32.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: local primary=1 2023-09-30T01:23:32.155 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: get_not_primary rbd GROUP 2023-09-30T01:23:32.155 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2023-09-30T01:23:32.155 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=GROUP 2023-09-30T01:23:32.156 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd GROUP 2023-09-30T01:23:32.156 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2023-09-30T01:23:32.156 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2023-09-30T01:23:32.157 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2023-09-30T01:23:32.157 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2023-09-30T01:23:32.561 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2023-09-30T01:23:32.561 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd GROUP 2023-09-30T01:23:32.561 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2023-09-30T01:23:32.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: local not_primary=0 2023-09-30T01:23:32.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: test_get_not_primary: test 0 '!=' 1 2023-09-30T01:23:32.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: test_get_not_primary: test 0 = 0 -o 0 = 1 2023-09-30T01:23:32.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: test_get_not_primary: teardown td/ceph-helpers 2023-09-30T01:23:32.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:23:32.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:23:32.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:23:32.938 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:23:32.938 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:23:32.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:23:32.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:23:32.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:23:33.051 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:23:33.051 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:23:33.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:23:33.054 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:23:33.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:23:33.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:23:33.056 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:23:33.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:23:33.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:23:33.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:23:33.059 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:23:33.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:23:33.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:23:33.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:23:33.079 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:23:33.079 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:23:33.079 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:23:33.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:23:33.082 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:23:33.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:23:33.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:23:33.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_num_active_clean td/ceph-helpers 2023-09-30T01:23:33.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: local dir=td/ceph-helpers 2023-09-30T01:23:33.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1374: test_get_num_active_clean: setup td/ceph-helpers 2023-09-30T01:23:33.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:23:33.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:23:33.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:23:33.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:23:33.083 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:23:33.083 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:23:33.084 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:23:33.084 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:23:33.084 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:23:33.084 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:23:33.086 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:23:33.087 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:23:33.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:23:33.089 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:23:33.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:23:33.090 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:23:33.095 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:23:33.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:23:33.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:23:33.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:23:33.095 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:23:33.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:23:33.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:23:33.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:23:33.097 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:23:33.098 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:23:33.098 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:23:33.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:23:33.099 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:23:33.099 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:23:33.099 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:23:33.102 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:23:33.102 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:23:33.102 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:23:33.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:23:33.104 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:23:33.105 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:23:33.105 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:23:33.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:23:33.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1375: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:23:33.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:23:33.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:23:33.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:23:33.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:23:33.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:23:33.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:23:33.476 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:23:33.476 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:23:33.476 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:23:33.477 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:23:33.477 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:23:33.477 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:23:33.477 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:23:33.478 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:23:33.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:23:33.784 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:23:33.784 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:23:33.784 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:23:33.784 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:23:33.785 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:23:33.785 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:23:33.785 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:23:33.785 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:23:33.786 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:23:33.786 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:23:33.786 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:23:33.787 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:23:33.787 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:23:33.787 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:23:33.872 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:23:33.872 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:23:33.872 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:23:33.872 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:23:33.873 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:23:33.873 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:23:33.873 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:23:33.874 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:23:33.874 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:23:33.874 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:23:33.874 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:23:33.874 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:23:33.874 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:23:33.874 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:23:33.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1376: test_get_num_active_clean: run_mgr td/ceph-helpers x 2023-09-30T01:23:33.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:23:33.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:23:33.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:23:33.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:23:33.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:23:33.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:23:34.858 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:23:34.858 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:23:34.858 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:23:34.859 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:23:34.859 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:23:34.859 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:23:34.859 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:23:34.860 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:23:34.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:23:34.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1377: test_get_num_active_clean: run_osd td/ceph-helpers 0 2023-09-30T01:23:34.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:23:34.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:23:34.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:23:34.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:23:34.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:23:34.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:23:34.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:23:34.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:23:34.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:23:34.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:23:34.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:23:34.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:23:34.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:23:34.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:23:34.899 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:23:34.899 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:23:34.899 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:23:34.900 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:23:34.901 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:23:34.901 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:23:34.901 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:23:34.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:23:34.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:23:34.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:23:34.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:23:34.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:23:34.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:23:34.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:23:34.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:23:34.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:23:34.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:23:34.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:23:34.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:23:34.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:23:34.903 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:23:34.904 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f2818031-395f-4180-896b-d1ed8d5629f0 2023-09-30T01:23:34.904 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 f2818031-395f-4180-896b-d1ed8d5629f0 2023-09-30T01:23:34.904 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f2818031-395f-4180-896b-d1ed8d5629f0' 2023-09-30T01:23:34.905 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:23:34.922 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCWeBdlBwbtNhAAx6sBAN0xYsL5kXR7NBaxTg== 2023-09-30T01:23:34.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCWeBdlBwbtNhAAx6sBAN0xYsL5kXR7NBaxTg=="}' 2023-09-30T01:23:34.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f2818031-395f-4180-896b-d1ed8d5629f0 -i td/ceph-helpers/0/new.json 2023-09-30T01:23:35.284 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:23:35.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:23:35.301 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWeBdlBwbtNhAAx6sBAN0xYsL5kXR7NBaxTg== --osd-uuid f2818031-395f-4180-896b-d1ed8d5629f0 2023-09-30T01:23:35.330 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:35.326+0000 7f02caeca3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:23:35.333 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:35.330+0000 7f02caeca3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:23:35.334 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:35.330+0000 7f02caeca3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:23:35.335 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:35.330+0000 7f02caeca3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:23:35.335 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:35.330+0000 7f02caeca3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:23:35.335 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:35.330+0000 7f02caeca3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:23:35.432 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:35.426+0000 7f02caeca3c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:23:41.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:23:41.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:23:41.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:23:41.804 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:23:41.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:23:42.361 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:23:42.362 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:23:42.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:23:42.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:23:42.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:23:42.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:23:42.393 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:42.390+0000 7fdcc16453c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:23:42.422 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:42.418+0000 7fdcc16453c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:23:42.424 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:42.422+0000 7fdcc16453c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:23:42.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:23:42.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:23:42.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:23:42.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:23:42.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:23:42.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:42.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:23:42.727 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:23:42.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:42.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:42.963 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:42.958+0000 7fdcc16453c0 -1 Falling back to public interface 2023-09-30T01:23:43.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:44.132 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:44.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:44.133 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:23:44.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:23:44.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:44.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:44.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:45.507 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:23:45.502+0000 7fdcc16453c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:23:45.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:45.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:45.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:23:45.519 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:23:45.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:45.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:45.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:46.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:46.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:46.972 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:23:46.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:23:46.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:46.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:47.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:48.393 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:48.393 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:48.394 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:23:48.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:23:48.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:48.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:48.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:49.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:49.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:49.818 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:23:49.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:23:49.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:49.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:50.355 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:51.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:51.357 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:51.357 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:23:51.357 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:23:51.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:51.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:51.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:52.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:52.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:52.769 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:23:52.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:23:52.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:52.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:53.256 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:54.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:54.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:54.259 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:23:54.259 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:23:54.259 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:54.259 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:54.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:55.687 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:55.687 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:55.687 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:23:55.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:23:55.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:55.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:56.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:57.259 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:57.259 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:57.259 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:23:57.259 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:23:57.260 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:57.260 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:57.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:23:58.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:23:58.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:23:58.957 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:23:58.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:23:58.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:23:58.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:23:59.564 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:00.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:00.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:00.566 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:24:00.567 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:24:00.567 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:00.567 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:01.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:02.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:02.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:02.495 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:24:02.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:24:02.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:02.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:03.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:04.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:04.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:04.192 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:24:04.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:24:04.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:04.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:04.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:05.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:05.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:05.792 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:24:05.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:24:05.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:05.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:06.892 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:07.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:07.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:07.894 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:24:07.895 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:24:07.895 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:07.895 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:08.428 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/99395,v1:127.0.0.1:6803/99395] [v2:127.0.0.1:6804/99395,v1:127.0.0.1:6805/99395] exists,up f2818031-395f-4180-896b-d1ed8d5629f0 2023-09-30T01:24:08.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:24:08.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:24:08.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:24:08.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1378: test_get_num_active_clean: create_rbd_pool 2023-09-30T01:24:08.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:24:08.858 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:24:08.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:24:08.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:24:09.682 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:24:09.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:24:10.700 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:24:11.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1379: test_get_num_active_clean: wait_for_clean 2023-09-30T01:24:11.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:24:11.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:24:11.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:24:11.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:24:11.852 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:24:11.852 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:24:11.852 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:24:11.853 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:24:11.853 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:24:11.853 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:24:12.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:24:12.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:24:12.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:24:12.060 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:24:12.060 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:24:12.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:24:12.474 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:24:12.474 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:24:12.474 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:24:12.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836489 2023-09-30T01:24:12.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836489 2023-09-30T01:24:12.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836489' 2023-09-30T01:24:12.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:24:12.642 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836489 2023-09-30T01:24:12.642 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:24:12.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:24:12.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:24:12.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836489 2023-09-30T01:24:12.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836489 2023-09-30T01:24:12.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2023-09-30T01:24:12.646 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836489 2023-09-30T01:24:12.646 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:24:13.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836487 -lt 21474836489 2023-09-30T01:24:13.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:24:14.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:24:14.075 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:24:14.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836490 -lt 21474836489 2023-09-30T01:24:14.489 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:24:14.489 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:24:14.489 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:24:15.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:24:15.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:24:15.113 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:24:15.113 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:24:15.113 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:24:15.113 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:24:15.114 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:24:15.114 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:24:15.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:24:15.531 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:24:15.532 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:24:15.532 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:24:16.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:24:16.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:24:16.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:24:16.075 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: get_num_active_clean 2023-09-30T01:24:16.075 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:24:16.075 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:24:16.075 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:24:16.075 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:24:16.076 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:24:16.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: local num_active_clean=4 2023-09-30T01:24:16.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1381: test_get_num_active_clean: test 4 = 4 2023-09-30T01:24:16.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1382: test_get_num_active_clean: teardown td/ceph-helpers 2023-09-30T01:24:16.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:24:16.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:24:16.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:24:16.493 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:24:16.493 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:24:16.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:24:16.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:24:16.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:24:16.606 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:24:16.607 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:24:16.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:24:16.609 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:24:16.611 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:24:16.611 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:24:16.612 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:24:16.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:24:16.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:24:16.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:24:16.615 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:24:16.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:24:16.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:24:16.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:24:16.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:24:16.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:24:16.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:24:16.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:24:16.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:24:16.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_num_active_or_peered td/ceph-helpers 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: local dir=td/ceph-helpers 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1404: test_get_num_active_or_peered: setup td/ceph-helpers 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:24:16.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:24:16.636 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:24:16.636 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:24:16.637 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:24:16.638 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:24:16.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:24:16.640 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:24:16.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:24:16.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:24:16.643 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:24:16.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:24:16.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:24:16.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:24:16.646 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:24:16.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:24:16.649 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:24:16.649 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:24:16.652 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:24:16.653 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:24:16.653 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:24:16.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:24:16.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:24:16.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:24:16.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:24:16.656 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:24:16.656 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:24:16.656 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:24:16.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:24:16.659 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:24:16.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:24:16.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:24:16.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:24:16.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1405: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:24:16.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:24:16.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:24:16.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:24:16.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:24:16.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:24:16.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:24:17.044 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:24:17.044 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:24:17.044 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:24:17.045 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:24:17.045 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:24:17.045 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:24:17.045 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:24:17.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:24:17.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:24:17.360 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:24:17.360 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:24:17.360 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:24:17.360 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:24:17.361 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:24:17.361 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:24:17.362 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:24:17.362 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:24:17.362 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:24:17.363 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:24:17.363 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:24:17.363 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:24:17.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:24:17.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:24:17.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:24:17.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:24:17.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:24:17.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:24:17.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:24:17.459 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:24:17.460 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:24:17.460 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:24:17.460 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:24:17.460 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:24:17.460 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:24:17.460 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:24:17.460 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:24:17.460 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:24:17.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1406: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2023-09-30T01:24:17.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:24:17.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:24:17.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:24:17.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:24:17.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:24:17.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:24:18.450 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:24:18.451 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:24:18.451 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:24:18.451 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:24:18.451 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:24:18.452 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:24:18.452 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:24:18.452 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:24:18.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:24:18.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1407: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2023-09-30T01:24:18.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:24:18.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:24:18.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:24:18.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:24:18.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:24:18.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:24:18.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:24:18.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:24:18.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:24:18.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:24:18.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:24:18.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:24:18.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:24:18.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:24:18.491 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:24:18.491 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:24:18.491 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:24:18.491 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:24:18.491 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:24:18.491 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:24:18.497 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:24:18.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:24:18.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:24:18.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:24:18.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:24:18.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:24:18.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:24:18.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:24:18.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:24:18.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:24:18.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:24:18.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:24:18.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:24:18.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:24:18.498 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:24:18.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=372d9d49-3f13-4ee6-acba-687a46a759fb 2023-09-30T01:24:18.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 372d9d49-3f13-4ee6-acba-687a46a759fb' 2023-09-30T01:24:18.499 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 372d9d49-3f13-4ee6-acba-687a46a759fb 2023-09-30T01:24:18.499 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:24:18.514 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCeBdl5kyNHhAAIsgNMfAEUyzz9tLjZ78y0A== 2023-09-30T01:24:18.514 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCeBdl5kyNHhAAIsgNMfAEUyzz9tLjZ78y0A=="}' 2023-09-30T01:24:18.514 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 372d9d49-3f13-4ee6-acba-687a46a759fb -i td/ceph-helpers/0/new.json 2023-09-30T01:24:18.876 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:24:18.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:24:18.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCeBdl5kyNHhAAIsgNMfAEUyzz9tLjZ78y0A== --osd-uuid 372d9d49-3f13-4ee6-acba-687a46a759fb 2023-09-30T01:24:18.922 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:18.917+0000 7f71b8ec73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:24:18.925 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:18.921+0000 7f71b8ec73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:24:18.927 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:18.921+0000 7f71b8ec73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:24:18.927 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:18.921+0000 7f71b8ec73c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:24:18.927 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:18.921+0000 7f71b8ec73c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:24:18.928 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:18.921+0000 7f71b8ec73c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:24:18.999 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:18.993+0000 7f71b8ec73c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:24:25.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:24:25.362 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:24:25.364 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:24:25.364 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:24:25.364 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:24:25.921 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:24:25.922 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:24:25.922 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:24:25.922 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:24:25.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:24:25.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:24:25.951 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:25.945+0000 7f659e8de3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:24:25.953 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:25.949+0000 7f659e8de3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:24:25.960 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:25.957+0000 7f659e8de3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:24:26.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:24:26.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:24:26.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:24:26.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:24:26.333 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:24:26.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:24:26.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:26.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:24:26.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:26.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:26.527 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:26.521+0000 7f659e8de3c0 -1 Falling back to public interface 2023-09-30T01:24:26.763 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:27.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:27.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:27.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:24:27.765 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:24:27.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:27.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:28.171 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:29.065 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:24:29.061+0000 7f659e8de3c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:24:29.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:29.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:29.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:24:29.174 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:24:29.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:29.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:29.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:30.714 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:30.714 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:30.714 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:24:30.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:24:30.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:30.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:31.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:32.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:32.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:32.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:24:32.127 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:24:32.127 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:32.127 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:32.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:33.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:33.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:33.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:24:33.619 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:24:33.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:33.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:34.055 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:35.057 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:35.057 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:35.057 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:24:35.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:24:35.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:35.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:35.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:36.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:36.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:36.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:24:36.526 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:24:36.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:36.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:37.099 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:38.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:38.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:38.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:24:38.102 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:24:38.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:38.103 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:38.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:39.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:39.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:39.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:24:39.549 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:24:39.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:39.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:40.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:41.055 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:41.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:41.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:24:41.056 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:24:41.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:41.057 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:41.500 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:42.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:42.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:42.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:24:42.502 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:24:42.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:42.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:43.035 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:44.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:44.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:44.038 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:24:44.038 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:24:44.038 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:44.038 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:45.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:46.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:46.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:46.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:24:46.008 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:24:46.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:46.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:46.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:47.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:47.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:47.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:24:47.806 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:24:47.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:47.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:48.521 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:49.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:49.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:49.523 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:24:49.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:24:49.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:49.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:50.355 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:24:51.357 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:24:51.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:24:51.358 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:24:51.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:24:51.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:24:51.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:24:51.781 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/101815,v1:127.0.0.1:6803/101815] [v2:127.0.0.1:6804/101815,v1:127.0.0.1:6805/101815] exists,up 372d9d49-3f13-4ee6-acba-687a46a759fb 2023-09-30T01:24:51.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:24:51.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:24:51.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:24:51.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1408: test_get_num_active_or_peered: create_rbd_pool 2023-09-30T01:24:51.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:24:52.220 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:24:52.230 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:24:52.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:24:52.868 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:24:52.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:24:53.877 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:24:54.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1409: test_get_num_active_or_peered: wait_for_clean 2023-09-30T01:24:54.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:24:54.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:24:54.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:24:54.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:24:54.905 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:24:54.906 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:24:54.906 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:24:54.907 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:24:54.907 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:24:54.907 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:24:55.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:24:55.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:24:55.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:24:55.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:24:55.115 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:24:55.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:24:55.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:24:55.580 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:24:55.580 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:24:55.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836494 2023-09-30T01:24:55.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836494 2023-09-30T01:24:55.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836494' 2023-09-30T01:24:55.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:24:55.740 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836494 2023-09-30T01:24:55.740 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:24:55.741 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:24:55.742 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836494 2023-09-30T01:24:55.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:24:55.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836494 2023-09-30T01:24:55.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2023-09-30T01:24:55.745 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836494 2023-09-30T01:24:55.745 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:24:56.141 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836492 -lt 21474836494 2023-09-30T01:24:56.141 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:24:57.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:24:57.143 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:24:57.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836492 -lt 21474836494 2023-09-30T01:24:57.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:24:58.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 299 -eq 0 ']' 2023-09-30T01:24:58.556 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:24:59.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836495 -lt 21474836494 2023-09-30T01:24:59.016 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:24:59.017 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:24:59.017 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:24:59.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:24:59.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:24:59.534 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:24:59.534 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:24:59.534 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:24:59.534 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:24:59.535 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:24:59.535 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:24:59.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:24:59.899 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:24:59.900 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:24:59.900 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:25:00.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:25:00.394 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:25:00.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:25:00.395 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: get_num_active_or_peered 2023-09-30T01:25:00.395 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2023-09-30T01:25:00.395 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2023-09-30T01:25:00.396 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2023-09-30T01:25:00.396 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2023-09-30T01:25:00.396 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2023-09-30T01:25:00.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: local num_peered=4 2023-09-30T01:25:00.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1411: test_get_num_active_or_peered: test 4 = 4 2023-09-30T01:25:00.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1412: test_get_num_active_or_peered: teardown td/ceph-helpers 2023-09-30T01:25:00.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:25:00.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:25:00.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:25:00.827 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:25:00.827 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:25:00.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:25:00.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:25:00.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:25:00.939 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:25:00.940 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:25:00.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:25:00.942 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:25:00.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:25:00.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:25:00.945 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:25:00.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:25:00.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:25:00.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:25:00.948 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:25:00.950 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:25:00.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:25:00.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:25:00.961 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:25:00.961 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:00.961 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:00.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:25:00.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:25:00.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:25:00.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:25:00.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_num_pgs td/ceph-helpers 2023-09-30T01:25:00.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: local dir=td/ceph-helpers 2023-09-30T01:25:00.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1431: test_get_num_pgs: setup td/ceph-helpers 2023-09-30T01:25:00.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:25:00.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:25:00.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:25:00.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:25:00.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:25:00.965 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:25:00.966 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:25:00.966 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:25:00.966 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:25:00.966 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:25:00.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:25:00.969 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:25:00.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:25:00.971 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:25:00.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:25:00.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:25:00.973 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:25:00.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:25:00.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:25:00.975 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:25:00.976 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:25:00.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:25:00.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:25:00.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:25:00.981 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:25:00.982 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:00.982 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:00.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:25:00.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:25:00.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:25:00.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:25:00.986 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:25:00.986 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:00.986 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:00.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:25:00.989 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:25:00.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:25:00.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:25:00.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:25:00.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:25:00.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:25:00.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:25:00.990 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:25:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:25:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:25:00.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:25:01.397 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:25:01.397 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:25:01.397 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:25:01.397 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:25:01.397 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:01.398 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:01.399 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:25:01.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:25:01.762 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:25:01.763 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:25:01.763 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:25:01.763 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:25:01.763 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:25:01.764 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:25:01.764 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:25:01.764 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:25:01.765 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:25:01.765 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:25:01.765 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:01.766 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:01.766 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:25:01.766 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:25:01.766 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:25:01.854 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:25:01.855 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:25:01.857 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:25:01.857 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:25:01.858 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:25:01.858 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:25:01.858 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:25:01.860 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:25:01.860 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:25:01.860 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:01.860 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:01.860 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:25:01.860 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:25:01.861 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:25:01.940 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: test_get_num_pgs: run_mgr td/ceph-helpers x 2023-09-30T01:25:01.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:25:01.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:25:01.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:25:01.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:25:01.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:25:01.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:25:02.849 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:25:02.849 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:25:02.849 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:25:02.849 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:25:02.849 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:02.850 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:02.850 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:25:02.851 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:25:02.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:25:02.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: test_get_num_pgs: run_osd td/ceph-helpers 0 2023-09-30T01:25:02.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:25:02.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:25:02.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:25:02.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:25:02.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:25:02.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:25:02.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:25:02.888 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:25:02.888 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:25:02.888 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:25:02.888 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:25:02.888 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:25:02.888 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:25:02.888 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:25:02.888 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:25:02.888 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:25:02.888 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:25:02.888 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:25:02.889 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:02.889 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:02.889 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:25:02.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:25:02.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:25:02.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:25:02.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:25:02.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:25:02.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:25:02.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:25:02.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:25:02.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:25:02.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:25:02.892 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:25:02.892 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:25:02.892 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:25:02.892 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:25:02.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7091f004-777b-4a69-927b-98c202254fec 2023-09-30T01:25:02.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7091f004-777b-4a69-927b-98c202254fec' 2023-09-30T01:25:02.893 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 7091f004-777b-4a69-927b-98c202254fec 2023-09-30T01:25:02.893 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:25:02.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDueBdl6sk+NhAAA9ETT58oFzlRVYf7+/AcNw== 2023-09-30T01:25:02.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDueBdl6sk+NhAAA9ETT58oFzlRVYf7+/AcNw=="}' 2023-09-30T01:25:02.913 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7091f004-777b-4a69-927b-98c202254fec -i td/ceph-helpers/0/new.json 2023-09-30T01:25:03.272 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:25:03.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:25:03.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDueBdl6sk+NhAAA9ETT58oFzlRVYf7+/AcNw== --osd-uuid 7091f004-777b-4a69-927b-98c202254fec 2023-09-30T01:25:03.321 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:03.316+0000 7f3e1584c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:03.323 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:03.316+0000 7f3e1584c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:03.324 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:03.320+0000 7f3e1584c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:03.325 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:03.320+0000 7f3e1584c3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:25:03.325 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:03.320+0000 7f3e1584c3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:25:03.326 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:03.320+0000 7f3e1584c3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:25:03.403 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:03.396+0000 7f3e1584c3c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:25:09.867 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:25:09.867 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:25:09.869 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:25:09.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:25:09.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:25:10.442 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:25:10.442 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:25:10.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:25:10.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:25:10.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:25:10.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:25:10.473 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:10.468+0000 7f2d1ba743c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:10.478 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:10.472+0000 7f2d1ba743c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:10.479 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:10.472+0000 7f2d1ba743c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:10.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:25:10.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:25:10.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:25:10.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:25:10.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:25:10.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:10.844 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:25:10.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:25:10.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:10.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:11.026 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:11.020+0000 7f2d1ba743c0 -1 Falling back to public interface 2023-09-30T01:25:11.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:12.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:12.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:12.283 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:25:12.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:25:12.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:12.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:12.677 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:13.656 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:13.648+0000 7f2d1ba743c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:25:13.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:13.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:13.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:25:13.679 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:25:13.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:13.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:14.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:15.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:15.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:15.216 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:25:15.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:25:15.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:15.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:15.705 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:16.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:16.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:16.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:25:16.707 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:25:16.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:16.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:17.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:18.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:18.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:18.133 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:25:18.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:25:18.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:18.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:18.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:19.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:19.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:19.675 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:25:19.675 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:25:19.675 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:19.675 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:20.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:21.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:21.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:21.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:25:21.102 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:25:21.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:21.103 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:21.504 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:22.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:22.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:22.506 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:25:22.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:25:22.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:22.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:22.933 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:23.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:23.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:23.935 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:25:23.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:25:23.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:23.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:24.587 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:25.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:25.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:25.590 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:25:25.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:25:25.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:25.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:26.142 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:27.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:27.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:27.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:25:27.144 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:25:27.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:27.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:27.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:28.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:28.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:28.739 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:25:28.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:25:28.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:28.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:29.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:30.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:30.685 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:30.685 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:25:30.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:25:30.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:30.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:31.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:32.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:32.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:32.512 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:25:32.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:25:32.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:32.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:33.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:34.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:34.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:34.192 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:25:34.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:25:34.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:34.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:34.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:35.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:35.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:35.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:25:35.918 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:25:35.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:35.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:36.494 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/104265,v1:127.0.0.1:6803/104265] [v2:127.0.0.1:6804/104265,v1:127.0.0.1:6805/104265] exists,up 7091f004-777b-4a69-927b-98c202254fec 2023-09-30T01:25:36.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:25:36.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:25:36.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:25:36.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: test_get_num_pgs: create_rbd_pool 2023-09-30T01:25:36.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:25:36.900 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:25:36.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:25:36.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:25:37.631 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:25:37.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:25:38.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:25:39.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1436: test_get_num_pgs: wait_for_clean 2023-09-30T01:25:39.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:25:39.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:25:39.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:25:39.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:25:39.691 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:25:39.692 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:25:39.692 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:25:39.692 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:25:39.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:25:39.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:25:39.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:25:39.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:25:39.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:25:39.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:25:39.899 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:25:40.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:25:40.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:25:40.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:25:40.354 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:25:40.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836490 2023-09-30T01:25:40.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836490 2023-09-30T01:25:40.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836490' 2023-09-30T01:25:40.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:25:40.503 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836490 2023-09-30T01:25:40.503 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:25:40.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:25:40.506 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836490 2023-09-30T01:25:40.507 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:25:40.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836490 2023-09-30T01:25:40.508 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836490 2023-09-30T01:25:40.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2023-09-30T01:25:40.508 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:25:40.915 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836488 -lt 21474836490 2023-09-30T01:25:40.915 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:25:41.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:25:41.918 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:25:42.366 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836491 -lt 21474836490 2023-09-30T01:25:42.367 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:25:42.367 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:25:42.368 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:25:42.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:25:42.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:25:42.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:25:42.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:25:42.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:25:42.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:25:42.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:25:42.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:25:43.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:25:43.245 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:25:43.245 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:25:43.245 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:25:43.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:25:43.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:25:43.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:25:43.731 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: get_num_pgs 2023-09-30T01:25:43.731 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:25:43.732 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:25:44.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: local num_pgs=4 2023-09-30T01:25:44.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1438: test_get_num_pgs: test 4 -gt 0 2023-09-30T01:25:44.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1439: test_get_num_pgs: teardown td/ceph-helpers 2023-09-30T01:25:44.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:25:44.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:25:44.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:25:44.328 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:25:44.328 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:25:44.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:25:44.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:25:44.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:25:44.440 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:25:44.440 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:25:44.442 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:25:44.443 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:25:44.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:25:44.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:25:44.445 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:25:44.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:25:44.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:25:44.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:25:44.449 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:25:44.450 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:25:44.452 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:25:44.452 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:25:44.464 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:25:44.464 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:44.464 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:44.464 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:25:44.466 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_op_scheduler td/ceph-helpers 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: test_get_op_scheduler: local dir=td/ceph-helpers 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: test_get_op_scheduler: setup td/ceph-helpers 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:25:44.467 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:25:44.468 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:25:44.468 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:25:44.468 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:25:44.471 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:25:44.471 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:25:44.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:25:44.474 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:25:44.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:25:44.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:25:44.476 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:25:44.477 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:25:44.477 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:25:44.478 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:25:44.478 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:25:44.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:25:44.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:25:44.481 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:25:44.483 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:25:44.483 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:44.483 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:44.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:25:44.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:25:44.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:25:44.486 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:25:44.487 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:25:44.488 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:44.488 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:44.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:25:44.489 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:25:44.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:25:44.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:25:44.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:25:44.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2196: test_get_op_scheduler: run_mon td/ceph-helpers a 2023-09-30T01:25:44.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:25:44.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:25:44.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:25:44.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:25:44.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:25:44.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2023-09-30T01:25:44.855 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:25:44.856 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:25:44.856 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:25:44.856 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:25:44.856 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:44.856 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:44.857 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:25:44.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:25:45.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:25:45.154 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:25:45.154 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:25:45.154 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:25:45.154 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:25:45.155 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:25:45.156 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:25:45.156 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:25:45.156 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:25:45.156 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:25:45.157 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:45.157 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:45.157 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:25:45.158 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:25:45.158 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:25:45.247 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:25:45.247 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:25:45.247 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:25:45.247 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:25:45.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:25:45.250 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:25:45.250 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:25:45.250 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:25:45.250 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:25:45.250 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:45.250 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:45.251 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:25:45.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:25:45.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:25:45.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: test_get_op_scheduler: run_mgr td/ceph-helpers x 2023-09-30T01:25:45.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:25:45.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:25:45.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:25:45.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:25:45.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:25:45.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:25:46.268 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:25:46.284 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:25:46.284 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:25:46.285 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:25:46.285 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:46.285 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:46.285 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:25:46.285 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:25:46.285 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:25:46.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2023-09-30T01:25:46.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:25:46.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:25:46.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:25:46.310 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:25:46.310 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:25:46.310 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:25:46.311 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:25:46.312 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:46.312 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:46.312 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:25:46.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:25:46.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:25:46.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:25:46.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:25:46.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:25:46.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:25:46.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:25:46.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:25:46.313 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:25:46.313 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:25:46.313 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:25:46.313 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=wpq 2023-09-30T01:25:46.313 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:25:46.313 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:25:46.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6dbdd4a5-ee0a-4837-8fb5-242122580814 2023-09-30T01:25:46.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6dbdd4a5-ee0a-4837-8fb5-242122580814' 2023-09-30T01:25:46.314 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 6dbdd4a5-ee0a-4837-8fb5-242122580814 2023-09-30T01:25:46.314 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:25:46.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaeRdlxryxExAAYTdGnIM0HAuaZWl2k/Px6g== 2023-09-30T01:25:46.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaeRdlxryxExAAYTdGnIM0HAuaZWl2k/Px6g=="}' 2023-09-30T01:25:46.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6dbdd4a5-ee0a-4837-8fb5-242122580814 -i td/ceph-helpers/0/new.json 2023-09-30T01:25:46.721 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:25:46.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:25:46.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQAaeRdlxryxExAAYTdGnIM0HAuaZWl2k/Px6g== --osd-uuid 6dbdd4a5-ee0a-4837-8fb5-242122580814 2023-09-30T01:25:46.769 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:46.767+0000 7f0a9b7dd3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:46.771 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:46.767+0000 7f0a9b7dd3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:46.773 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:46.771+0000 7f0a9b7dd3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:46.773 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:46.771+0000 7f0a9b7dd3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:25:46.774 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:46.771+0000 7f0a9b7dd3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:25:46.774 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:46.771+0000 7f0a9b7dd3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:25:46.885 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:46.883+0000 7f0a9b7dd3c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:25:53.295 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:25:53.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:25:53.298 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:25:53.298 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:25:53.298 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:25:53.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:25:53.814 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:25:53.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2023-09-30T01:25:53.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:25:53.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:25:53.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:25:53.843 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:53.839+0000 7f66842363c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:53.861 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:53.859+0000 7f66842363c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:53.862 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:53.859+0000 7f66842363c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:54.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:25:54.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:25:54.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:25:54.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:25:54.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:25:54.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:54.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:25:54.227 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:25:54.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:54.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:54.419 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:54.415+0000 7f66842363c0 -1 Falling back to public interface 2023-09-30T01:25:54.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:55.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:55.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:55.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:25:55.669 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:25:55.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:55.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:56.075 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:56.964 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:56.959+0000 7f66842363c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:25:57.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:57.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:57.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:25:57.077 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:25:57.078 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:57.078 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:57.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:25:58.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:25:58.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:25:58.581 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:25:58.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:25:58.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:25:58.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:25:58.991 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/106688,v1:127.0.0.1:6803/106688] [v2:127.0.0.1:6804/106688,v1:127.0.0.1:6805/106688] exists,up 6dbdd4a5-ee0a-4837-8fb5-242122580814 2023-09-30T01:25:58.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:25:58.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:25:58.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:25:58.993 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: test_get_op_scheduler: get_op_scheduler 0 2023-09-30T01:25:58.993 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: get_op_scheduler: local id=0 2023-09-30T01:25:58.993 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: get_op_scheduler: get_config osd 0 osd_op_queue 2023-09-30T01:25:58.993 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2023-09-30T01:25:58.993 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2023-09-30T01:25:58.993 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2023-09-30T01:25:58.994 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2023-09-30T01:25:58.994 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2023-09-30T01:25:58.994 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:25:58.994 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:25:58.995 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:25:58.995 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:58.995 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:58.995 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.0.asok 2023-09-30T01:25:58.995 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:25:58.995 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-osd.0.asok config get osd_op_queue 2023-09-30T01:25:59.076 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: test_get_op_scheduler: test wpq = wpq 2023-09-30T01:25:59.076 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2023-09-30T01:25:59.076 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:25:59.076 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:25:59.076 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-09-30T01:25:59.076 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:25:59.076 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2023-09-30T01:25:59.076 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:25:59.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:25:59.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:25:59.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:25:59.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2023-09-30T01:25:59.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2023-09-30T01:25:59.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:25:59.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:25:59.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:25:59.077 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:25:59.077 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:25:59.077 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:25:59.079 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:25:59.079 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:25:59.079 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:25:59.079 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:25:59.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:25:59.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:25:59.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:25:59.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:25:59.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:25:59.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:25:59.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:25:59.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:25:59.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:25:59.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:25:59.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:25:59.081 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2023-09-30T01:25:59.081 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2023-09-30T01:25:59.081 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:25:59.081 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa38548c-134e-49ec-b9fa-8d12a1f749f1 2023-09-30T01:25:59.081 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 aa38548c-134e-49ec-b9fa-8d12a1f749f1' 2023-09-30T01:25:59.082 INFO:tasks.workunit.client.0.smithi186.stdout:add osd1 aa38548c-134e-49ec-b9fa-8d12a1f749f1 2023-09-30T01:25:59.082 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:25:59.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAneRdl15GqBRAAHaxuKiWbOxvFdGw+DHuABQ== 2023-09-30T01:25:59.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAneRdl15GqBRAAHaxuKiWbOxvFdGw+DHuABQ=="}' 2023-09-30T01:25:59.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa38548c-134e-49ec-b9fa-8d12a1f749f1 -i td/ceph-helpers/1/new.json 2023-09-30T01:25:59.652 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:25:59.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2023-09-30T01:25:59.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAneRdl15GqBRAAHaxuKiWbOxvFdGw+DHuABQ== --osd-uuid aa38548c-134e-49ec-b9fa-8d12a1f749f1 2023-09-30T01:25:59.688 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:59.687+0000 7f8777da23c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:59.692 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:59.691+0000 7f8777da23c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:59.694 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:59.691+0000 7f8777da23c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:25:59.694 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:59.691+0000 7f8777da23c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:25:59.694 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:59.691+0000 7f8777da23c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:25:59.694 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:59.691+0000 7f8777da23c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:25:59.784 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:25:59.783+0000 7f8777da23c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2023-09-30T01:26:06.050 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2023-09-30T01:26:06.050 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:26:06.052 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd1 key to auth repository 2023-09-30T01:26:06.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-09-30T01:26:06.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:26:06.662 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:26:06.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-09-30T01:26:06.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2023-09-30T01:26:06.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:26:06.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:26:06.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:26:06.692 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:06.691+0000 7f66f08903c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:26:06.703 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:06.699+0000 7f66f08903c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:26:06.704 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:06.703+0000 7f66f08903c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:26:07.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-09-30T01:26:07.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:26:07.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:26:07.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:26:07.071 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:26:07.071 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:07.071 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:26:07.071 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:26:07.071 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:07.071 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:07.282 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:07.279+0000 7f66f08903c0 -1 Falling back to public interface 2023-09-30T01:26:07.497 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:08.499 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:08.499 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:08.499 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:26:08.499 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:26:08.500 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:08.500 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:08.915 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:09.890 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:09.887+0000 7f66f08903c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:26:09.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:09.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:09.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:26:09.918 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:26:09.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:09.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:10.587 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:11.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:11.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:11.589 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:26:11.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:26:11.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:11.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:12.003 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:13.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:13.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:13.004 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:26:13.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:26:13.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:13.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:13.419 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:14.421 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:14.421 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:14.422 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:26:14.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:26:14.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:14.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:14.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:15.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:15.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:15.882 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:26:15.882 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:26:15.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:15.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:16.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:17.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:17.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:17.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:26:17.353 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:26:17.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:17.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:17.870 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:18.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:18.873 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:18.873 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:26:18.873 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:26:18.874 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:18.874 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:19.320 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:20.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:20.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:20.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:26:20.322 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:26:20.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:20.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:20.775 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:21.777 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:21.777 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:21.777 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:26:21.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:26:21.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:21.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:22.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:23.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:23.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:23.535 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:26:23.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:26:23.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:23.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:24.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:25.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:25.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:25.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:26:25.290 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:26:25.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:25.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:26.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:27.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:27.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:27.329 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:26:27.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:26:27.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:27.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:28.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:29.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:29.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:29.135 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:26:29.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:26:29.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:29.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:29.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:30.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:30.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:30.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:26:30.988 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:26:30.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:30.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:31.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:32.832 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:32.833 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:32.833 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:26:32.833 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:26:32.833 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:32.834 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:26:33.289 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/108100,v1:127.0.0.1:6811/108100] [v2:127.0.0.1:6812/108100,v1:127.0.0.1:6813/108100] exists,up aa38548c-134e-49ec-b9fa-8d12a1f749f1 2023-09-30T01:26:33.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:26:33.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:26:33.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:26:33.290 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: test_get_op_scheduler: get_op_scheduler 1 2023-09-30T01:26:33.291 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: get_op_scheduler: local id=1 2023-09-30T01:26:33.291 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: get_op_scheduler: get_config osd 1 osd_op_queue 2023-09-30T01:26:33.291 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2023-09-30T01:26:33.291 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=1 2023-09-30T01:26:33.291 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2023-09-30T01:26:33.291 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2023-09-30T01:26:33.292 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.1 2023-09-30T01:26:33.295 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-09-30T01:26:33.295 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-09-30T01:26:33.295 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:26:33.295 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:26:33.295 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:26:33.295 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.1.asok 2023-09-30T01:26:33.296 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:26:33.296 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-osd.1.asok config get osd_op_queue 2023-09-30T01:26:33.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2023-09-30T01:26:33.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2204: test_get_op_scheduler: teardown td/ceph-helpers 2023-09-30T01:26:33.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:26:33.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:26:33.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:26:33.389 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:26:33.389 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:26:33.390 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:26:33.390 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:26:33.390 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:26:33.500 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:26:33.501 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:26:33.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:26:33.503 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:26:33.505 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:26:33.505 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:26:33.506 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:26:33.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:26:33.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:26:33.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:26:33.509 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:26:33.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:26:33.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:26:33.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:26:33.523 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:26:33.523 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:26:33.523 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:26:33.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:26:33.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:26:33.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:26:33.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:26:33.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2023-09-30T01:26:33.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2023-09-30T01:26:33.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1461: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2023-09-30T01:26:33.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:26:33.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:26:33.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:26:33.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:26:33.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:26:33.527 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:26:33.528 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:26:33.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:26:33.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:26:33.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:26:33.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:26:33.533 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:26:33.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:26:33.533 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:26:33.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:26:33.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:26:33.536 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:26:33.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:26:33.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:26:33.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:26:33.538 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:26:33.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:26:33.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:26:33.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:26:33.544 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:26:33.544 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:26:33.544 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:26:33.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:26:33.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:26:33.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:26:33.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:26:33.548 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:26:33.548 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:26:33.548 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:26:33.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:26:33.551 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:26:33.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:26:33.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:26:33.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:26:33.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1462: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:26:33.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:26:33.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:26:33.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:26:33.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:26:33.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:26:33.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:26:33.923 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:26:33.923 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:26:33.923 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:26:33.924 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:26:33.924 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:26:33.924 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:26:33.925 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:26:33.925 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:26:34.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:26:34.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:26:34.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:26:34.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:26:34.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:26:34.215 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:26:34.215 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:26:34.215 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:26:34.215 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:26:34.216 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:26:34.216 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:26:34.216 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:26:34.216 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:26:34.217 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:26:34.217 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:26:34.306 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:26:34.306 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:26:34.306 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:26:34.306 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:26:34.306 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:26:34.307 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:26:34.307 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:26:34.307 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:26:34.307 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:26:34.308 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:26:34.308 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:26:34.308 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:26:34.308 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:26:34.308 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:26:34.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1463: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2023-09-30T01:26:34.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:26:34.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:26:34.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:26:34.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:26:34.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:26:34.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:26:35.290 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:26:35.291 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:26:35.291 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:26:35.291 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:26:35.291 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:26:35.292 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:26:35.292 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:26:35.292 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:26:35.293 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:26:35.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1464: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2023-09-30T01:26:35.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:26:35.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:26:35.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:26:35.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:26:35.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:26:35.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:26:35.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:26:35.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:26:35.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:26:35.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:26:35.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:26:35.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:26:35.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:26:35.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:26:35.333 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:26:35.333 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:26:35.333 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:26:35.333 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:26:35.334 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:26:35.334 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:26:35.334 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:26:35.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:26:35.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:26:35.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:26:35.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:26:35.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:26:35.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:26:35.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:26:35.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:26:35.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:26:35.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:26:35.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:26:35.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:26:35.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:26:35.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:26:35.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50d21d3e-2340-4aed-a647-8862959c9561 2023-09-30T01:26:35.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 50d21d3e-2340-4aed-a647-8862959c9561' 2023-09-30T01:26:35.338 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 50d21d3e-2340-4aed-a647-8862959c9561 2023-09-30T01:26:35.338 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:26:35.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLeRdlCT9iFRAA+9CrYLHL/hS9CmxBn2KcJg== 2023-09-30T01:26:35.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLeRdlCT9iFRAA+9CrYLHL/hS9CmxBn2KcJg=="}' 2023-09-30T01:26:35.360 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50d21d3e-2340-4aed-a647-8862959c9561 -i td/ceph-helpers/0/new.json 2023-09-30T01:26:35.715 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:26:35.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:26:35.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLeRdlCT9iFRAA+9CrYLHL/hS9CmxBn2KcJg== --osd-uuid 50d21d3e-2340-4aed-a647-8862959c9561 2023-09-30T01:26:35.764 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:35.758+0000 7f1dcd13c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:26:35.766 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:35.762+0000 7f1dcd13c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:26:35.769 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:35.766+0000 7f1dcd13c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:26:35.770 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:35.766+0000 7f1dcd13c3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:26:35.770 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:35.766+0000 7f1dcd13c3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:26:35.770 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:35.766+0000 7f1dcd13c3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:26:35.829 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:35.826+0000 7f1dcd13c3c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:26:42.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:26:42.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:26:42.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:26:42.224 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:26:42.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:26:42.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:26:42.750 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:26:42.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:26:42.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:26:42.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:26:42.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:26:42.779 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:42.774+0000 7fcf9ca773c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:26:42.808 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:42.806+0000 7fcf9ca773c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:26:42.810 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:42.806+0000 7fcf9ca773c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:26:43.149 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:26:43.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:26:43.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:26:43.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:26:43.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:26:43.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:43.150 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:26:43.151 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:26:43.151 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:43.151 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:26:43.351 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:43.346+0000 7fcf9ca773c0 -1 Falling back to public interface 2023-09-30T01:26:43.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:44.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:44.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:44.543 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:26:44.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:26:44.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:44.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:26:44.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:45.909 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:26:45.906+0000 7fcf9ca773c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:26:45.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:45.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:45.950 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:26:45.951 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:26:45.951 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:45.951 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:26:46.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:47.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:47.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:47.526 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:26:47.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:26:47.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:47.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:26:47.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:48.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:48.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:48.979 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:26:48.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:26:48.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:48.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:26:49.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:50.477 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:50.477 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:50.477 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:26:50.478 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:26:50.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:50.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:26:50.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:51.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:51.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:51.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:26:51.886 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:26:51.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:51.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:26:52.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:53.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:53.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:53.330 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:26:53.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:26:53.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:53.331 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:26:53.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:54.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:54.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:54.777 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:26:54.777 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:26:54.777 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:54.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:26:55.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:56.195 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:56.195 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:56.195 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:26:56.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:26:56.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:56.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:26:56.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:57.717 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:57.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:57.718 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:26:57.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:26:57.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:57.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:26:58.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:26:59.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:26:59.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:26:59.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:26:59.310 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:26:59.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:26:59.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:00.396 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:01.398 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:01.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:01.399 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:27:01.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:27:01.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:01.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:02.534 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:03.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:03.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:03.536 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:27:03.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:27:03.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:03.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:04.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:05.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:05.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:05.645 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:27:05.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:27:05.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:05.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:06.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:07.459 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:07.459 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:07.459 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:27:07.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:27:07.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:07.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:08.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:09.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:09.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:09.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:27:09.009 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:27:09.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:09.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:09.467 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/110155,v1:127.0.0.1:6803/110155] [v2:127.0.0.1:6804/110155,v1:127.0.0.1:6805/110155] exists,up 50d21d3e-2340-4aed-a647-8862959c9561 2023-09-30T01:27:09.468 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:27:09.468 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:27:09.468 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:27:09.468 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1465: test_get_osd_id_used_by_pgs: create_rbd_pool 2023-09-30T01:27:09.469 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:27:09.948 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:27:09.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:27:09.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:27:10.678 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:27:10.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:27:11.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:27:12.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1466: test_get_osd_id_used_by_pgs: wait_for_clean 2023-09-30T01:27:12.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:27:12.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:27:12.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:27:12.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:27:12.829 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:27:12.830 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:27:12.830 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:27:12.830 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:27:12.831 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:27:12.831 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:27:13.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:27:13.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:27:13.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:27:13.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:27:13.040 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:27:13.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:27:13.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:27:13.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:27:13.444 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:27:13.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836486 2023-09-30T01:27:13.591 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836486 2023-09-30T01:27:13.591 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836486' 2023-09-30T01:27:13.591 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:27:13.591 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836486 2023-09-30T01:27:13.591 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:27:13.593 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:27:13.594 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:27:13.594 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836486 2023-09-30T01:27:13.594 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836486 2023-09-30T01:27:13.594 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2023-09-30T01:27:13.595 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836486 2023-09-30T01:27:13.595 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:27:14.003 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836484 -lt 21474836486 2023-09-30T01:27:14.003 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:27:15.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:27:15.006 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:27:15.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836487 -lt 21474836486 2023-09-30T01:27:15.423 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:27:15.423 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:27:15.424 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:27:15.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:27:15.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:27:15.980 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:27:15.980 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:27:15.980 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:27:15.980 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:27:15.981 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:27:15.981 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:27:16.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:27:16.403 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:27:16.404 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:27:16.404 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:27:16.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:27:16.931 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:27:16.931 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:27:16.932 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2023-09-30T01:27:16.932 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: uniq 2023-09-30T01:27:16.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2023-09-30T01:27:16.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2023-09-30T01:27:16.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2023-09-30T01:27:17.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: local osd_ids=0 2023-09-30T01:27:17.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: test_get_osd_id_used_by_pgs: test 0 = 0 2023-09-30T01:27:17.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2023-09-30T01:27:17.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:27:17.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:27:17.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:27:17.289 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:27:17.289 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:27:17.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:27:17.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:27:17.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:27:17.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:27:17.403 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:27:17.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:27:17.405 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:27:17.407 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:27:17.407 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:27:17.408 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:27:17.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:27:17.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:27:17.410 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:27:17.410 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:27:17.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:27:17.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:27:17.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:27:17.425 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:27:17.425 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:27:17.426 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:27:17.426 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:27:17.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:27:17.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:27:17.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:27:17.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_osds td/ceph-helpers 2023-09-30T01:27:17.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: local dir=td/ceph-helpers 2023-09-30T01:27:17.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: setup td/ceph-helpers 2023-09-30T01:27:17.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:27:17.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:27:17.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:27:17.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:27:17.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:27:17.431 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:27:17.431 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:27:17.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:27:17.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:27:17.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:27:17.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:27:17.434 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:27:17.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:27:17.436 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:27:17.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:27:17.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:27:17.438 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:27:17.440 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:27:17.440 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:27:17.441 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:27:17.441 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:27:17.442 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:27:17.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:27:17.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:27:17.446 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:27:17.446 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:27:17.446 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:27:17.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:27:17.450 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:27:17.450 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:27:17.450 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:27:17.450 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:27:17.450 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:27:17.450 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:27:17.450 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:27:17.452 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:27:17.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:27:17.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:27:17.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:27:17.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2023-09-30T01:27:17.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:27:17.454 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:27:17.454 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:27:17.454 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:27:17.454 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:27:17.454 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2023-09-30T01:27:17.850 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:27:17.851 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:27:17.851 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:27:17.851 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:27:17.851 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:27:17.852 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:27:17.852 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:27:17.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2023-09-30T01:27:18.138 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:27:18.139 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:27:18.140 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:27:18.140 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:27:18.140 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:27:18.142 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:27:18.142 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:27:18.142 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:27:18.143 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:27:18.143 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:27:18.143 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:27:18.143 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:27:18.144 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:27:18.144 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:27:18.144 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:27:18.236 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:27:18.236 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:27:18.236 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:27:18.236 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:27:18.237 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:27:18.237 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:27:18.237 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:27:18.237 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:27:18.238 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:27:18.238 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:27:18.238 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:27:18.238 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:27:18.238 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:27:18.238 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:27:18.321 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: test_get_osds: run_mgr td/ceph-helpers x 2023-09-30T01:27:18.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:27:18.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:27:18.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:27:18.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:27:18.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:27:18.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:27:19.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:27:19.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:27:19.214 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:27:19.214 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:27:19.215 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:27:19.215 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:27:19.215 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:27:19.216 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:27:19.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:27:19.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: test_get_osds: run_osd td/ceph-helpers 0 2023-09-30T01:27:19.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:27:19.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:27:19.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:27:19.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:27:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:27:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:27:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:27:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:27:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:27:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:27:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:27:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:27:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:27:19.251 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:27:19.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:27:19.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:27:19.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:27:19.251 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:27:19.251 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:27:19.251 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:27:19.252 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:27:19.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:27:19.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:27:19.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:27:19.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:27:19.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:27:19.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:27:19.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:27:19.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:27:19.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:27:19.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:27:19.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:27:19.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:27:19.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:27:19.254 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:27:19.255 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4022852b-3698-4302-a587-84d6d44d6662 2023-09-30T01:27:19.255 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4022852b-3698-4302-a587-84d6d44d6662' 2023-09-30T01:27:19.255 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 4022852b-3698-4302-a587-84d6d44d6662 2023-09-30T01:27:19.255 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:27:19.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3eRdluSNtEBAAInSPEr2ANga2kb1oMx2kFA== 2023-09-30T01:27:19.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3eRdluSNtEBAAInSPEr2ANga2kb1oMx2kFA=="}' 2023-09-30T01:27:19.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4022852b-3698-4302-a587-84d6d44d6662 -i td/ceph-helpers/0/new.json 2023-09-30T01:27:19.603 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:27:19.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:27:19.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3eRdluSNtEBAAInSPEr2ANga2kb1oMx2kFA== --osd-uuid 4022852b-3698-4302-a587-84d6d44d6662 2023-09-30T01:27:19.647 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:19.642+0000 7f9c25c833c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:27:19.664 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:19.662+0000 7f9c25c833c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:27:19.666 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:19.662+0000 7f9c25c833c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:27:19.667 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:19.662+0000 7f9c25c833c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:27:19.667 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:19.662+0000 7f9c25c833c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:27:19.667 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:19.662+0000 7f9c25c833c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:27:19.712 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:19.706+0000 7f9c25c833c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:27:26.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:27:26.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:27:26.224 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:27:26.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:27:26.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:27:26.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:27:26.786 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:27:26.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:27:26.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:27:26.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:27:26.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:27:26.808 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:26.801+0000 7ff7e4ed33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:27:26.810 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:26.805+0000 7ff7e4ed33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:27:26.812 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:26.805+0000 7ff7e4ed33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:27:27.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:27:27.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:27:27.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:27:27.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:27:27.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:27:27.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:27.176 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:27:27.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:27:27.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:27.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:27.394 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:27.389+0000 7ff7e4ed33c0 -1 Falling back to public interface 2023-09-30T01:27:27.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:28.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:28.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:28.667 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:27:28.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:27:28.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:28.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:29.067 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:29.997 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:29.993+0000 7ff7e4ed33c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:27:30.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:30.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:30.068 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:27:30.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:27:30.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:30.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:30.558 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:31.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:31.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:31.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:27:31.561 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:27:31.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:31.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:31.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:32.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:32.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:32.997 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:27:32.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:27:32.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:32.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:33.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:34.461 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:34.462 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:34.462 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:27:34.462 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:27:34.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:34.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:34.921 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:35.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:35.920 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:35.920 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:27:35.920 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:27:35.920 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:35.921 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:36.395 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:37.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:37.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:37.398 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:27:37.398 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:27:37.398 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:37.398 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:37.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:38.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:38.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:38.801 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:27:38.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:27:38.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:38.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:39.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:40.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:40.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:40.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:27:40.245 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:27:40.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:40.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:40.807 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:41.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:41.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:41.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:27:41.809 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:27:41.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:41.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:42.914 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:43.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:43.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:43.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:27:43.918 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:27:43.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:43.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:44.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:45.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:45.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:45.808 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:27:45.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:27:45.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:45.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:46.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:47.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:47.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:47.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:27:47.789 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:27:47.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:47.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:48.594 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:49.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:49.597 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:49.597 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:27:49.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:27:49.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:49.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:50.132 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:51.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:51.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:51.135 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:27:51.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:27:51.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:51.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:51.601 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:27:52.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:27:52.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:27:52.604 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:27:52.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:27:52.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:27:52.605 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:27:53.007 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/112580,v1:127.0.0.1:6803/112580] [v2:127.0.0.1:6804/112580,v1:127.0.0.1:6805/112580] exists,up 4022852b-3698-4302-a587-84d6d44d6662 2023-09-30T01:27:53.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:27:53.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:27:53.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:27:53.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1037: test_get_osds: run_osd td/ceph-helpers 1 2023-09-30T01:27:53.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:27:53.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:27:53.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-09-30T01:27:53.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:27:53.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2023-09-30T01:27:53.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:27:53.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:27:53.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:27:53.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:27:53.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2023-09-30T01:27:53.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2023-09-30T01:27:53.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:27:53.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:27:53.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:27:53.011 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:27:53.011 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:27:53.011 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:27:53.011 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:27:53.011 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:27:53.011 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:27:53.011 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:27:53.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:27:53.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:27:53.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:27:53.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:27:53.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:27:53.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:27:53.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:27:53.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:27:53.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:27:53.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:27:53.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:27:53.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:27:53.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2023-09-30T01:27:53.014 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:27:53.014 INFO:tasks.workunit.client.0.smithi186.stdout:add osd1 bc0f3d08-69bf-4e6c-ab21-00be440d781f 2023-09-30T01:27:53.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bc0f3d08-69bf-4e6c-ab21-00be440d781f 2023-09-30T01:27:53.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bc0f3d08-69bf-4e6c-ab21-00be440d781f' 2023-09-30T01:27:53.015 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:27:53.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZeRdlzL/qARAAqFSpL1Kl7G4nuSMoQXg01A== 2023-09-30T01:27:53.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZeRdlzL/qARAAqFSpL1Kl7G4nuSMoQXg01A=="}' 2023-09-30T01:27:53.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bc0f3d08-69bf-4e6c-ab21-00be440d781f -i td/ceph-helpers/1/new.json 2023-09-30T01:27:53.576 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:27:53.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2023-09-30T01:27:53.592 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZeRdlzL/qARAAqFSpL1Kl7G4nuSMoQXg01A== --osd-uuid bc0f3d08-69bf-4e6c-ab21-00be440d781f 2023-09-30T01:27:53.623 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:53.617+0000 7fdfbb0e73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:27:53.649 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:53.645+0000 7fdfbb0e73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:27:53.651 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:53.645+0000 7fdfbb0e73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:27:53.653 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:53.649+0000 7fdfbb0e73c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:27:53.653 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:53.649+0000 7fdfbb0e73c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:27:53.653 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:53.649+0000 7fdfbb0e73c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:27:53.741 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:27:53.737+0000 7fdfbb0e73c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2023-09-30T01:28:00.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2023-09-30T01:28:00.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:28:00.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-09-30T01:28:00.068 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd1 key to auth repository 2023-09-30T01:28:00.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:28:00.606 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:28:00.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-09-30T01:28:00.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:28:00.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:28:00.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:28:00.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:28:00.638 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:00.633+0000 7f30632d43c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:28:00.640 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:00.637+0000 7f30632d43c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:28:00.641 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:00.637+0000 7f30632d43c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:28:01.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-09-30T01:28:01.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:28:01.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:28:01.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:28:01.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:28:01.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:01.048 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:28:01.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:28:01.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:01.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:01.182 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:01.177+0000 7f30632d43c0 -1 Falling back to public interface 2023-09-30T01:28:01.551 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:02.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:02.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:02.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:28:02.554 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:28:02.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:02.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:02.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:03.759 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:03.753+0000 7f30632d43c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:28:03.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:03.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:03.959 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:28:03.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:28:03.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:03.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:04.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:05.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:05.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:05.484 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:28:05.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:28:05.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:05.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:05.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:06.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:06.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:06.893 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:28:06.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:28:06.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:06.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:07.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:08.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:08.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:08.308 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:28:08.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:28:08.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:08.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:08.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:09.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:09.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:09.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:28:09.963 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:28:09.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:09.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:10.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:11.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:11.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:11.369 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:28:11.369 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:28:11.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:11.370 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:11.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:12.871 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:12.871 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:12.872 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:28:12.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:28:12.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:12.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:13.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:14.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:14.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:14.307 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:28:14.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:28:14.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:14.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:14.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:15.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:15.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:15.758 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:28:15.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:28:15.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:15.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:16.286 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:17.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:17.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:17.288 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:28:17.288 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:28:17.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:17.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:18.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:19.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:19.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:19.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:28:19.095 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:28:19.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:19.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:19.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:20.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:20.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:20.729 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:28:20.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:28:20.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:20.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:21.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:22.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:22.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:22.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:28:22.619 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:28:22.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:22.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:23.379 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:24.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:24.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:24.382 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:28:24.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:28:24.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:24.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:25.223 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:26.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:26.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:26.225 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:28:26.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:28:26.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:26.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:26.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:27.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:27.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:27.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-09-30T01:28:27.654 INFO:tasks.workunit.client.0.smithi186.stdout:17 2023-09-30T01:28:27.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:27.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:28:28.039 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/114331,v1:127.0.0.1:6811/114331] [v2:127.0.0.1:6812/114331,v1:127.0.0.1:6813/114331] exists,up bc0f3d08-69bf-4e6c-ab21-00be440d781f 2023-09-30T01:28:28.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:28:28.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:28:28.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:28:28.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: test_get_osds: create_rbd_pool 2023-09-30T01:28:28.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:28:28.549 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:28:28.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:28:28.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:28:29.189 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:28:29.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:28:30.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:28:31.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1039: test_get_osds: wait_for_clean 2023-09-30T01:28:31.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:28:31.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:28:31.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:28:31.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:28:31.508 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:28:31.509 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:28:31.509 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:28:31.509 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:28:31.509 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:28:31.510 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:28:31.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:28:31.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:28:31.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:28:31.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:28:31.710 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:28:32.123 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2023-09-30T01:28:32.124 INFO:tasks.workunit.client.0.smithi186.stderr:1' 2023-09-30T01:28:32.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:28:32.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:28:32.124 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:28:32.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836497 2023-09-30T01:28:32.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836497 2023-09-30T01:28:32.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836497' 2023-09-30T01:28:32.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:28:32.304 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-09-30T01:28:32.454 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=42949672972 2023-09-30T01:28:32.454 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 42949672972 2023-09-30T01:28:32.454 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836497 1-42949672972' 2023-09-30T01:28:32.454 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:28:32.455 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836497 2023-09-30T01:28:32.455 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:28:32.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:28:32.457 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836497 2023-09-30T01:28:32.457 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:28:32.459 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836497 2023-09-30T01:28:32.459 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2023-09-30T01:28:32.459 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836497 2023-09-30T01:28:32.459 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:28:32.866 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836497 -lt 21474836497 2023-09-30T01:28:32.867 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:28:32.867 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-42949672972 2023-09-30T01:28:32.868 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:28:32.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2023-09-30T01:28:32.870 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:28:32.871 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-42949672972 2023-09-30T01:28:32.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=42949672972 2023-09-30T01:28:32.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2023-09-30T01:28:32.872 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.1 seq 42949672972 2023-09-30T01:28:32.873 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:28:33.295 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672970 -lt 42949672972 2023-09-30T01:28:33.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:28:34.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:28:34.297 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:28:34.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 42949672972 -lt 42949672972 2023-09-30T01:28:34.702 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:28:34.703 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:28:34.703 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:28:35.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:28:35.255 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:28:35.255 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:28:35.255 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:28:35.255 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:28:35.255 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:28:35.256 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:28:35.256 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:28:35.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:28:35.671 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:28:35.671 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:28:35.672 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:28:36.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:28:36.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:28:36.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:28:36.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1040: test_get_osds: create_rbd_pool 2023-09-30T01:28:36.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:28:36.867 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:28:36.874 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:28:36.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:28:37.557 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:28:37.568 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:28:38.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:28:39.922 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: get_osds rbd GROUP 2023-09-30T01:28:39.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=rbd 2023-09-30T01:28:39.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2023-09-30T01:28:39.923 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=GROUP 2023-09-30T01:28:39.924 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map rbd GROUP 2023-09-30T01:28:39.924 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2023-09-30T01:28:40.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2023-09-30T01:28:40.382 INFO:tasks.workunit.client.0.smithi186.stderr:0' 2023-09-30T01:28:40.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2023-09-30T01:28:40.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1042: test_get_osds: teardown td/ceph-helpers 2023-09-30T01:28:40.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:28:40.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:28:40.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:28:40.383 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:28:40.383 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:28:40.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:28:40.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:28:40.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:28:40.499 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:28:40.500 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:28:40.501 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:28:40.502 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:28:40.504 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:28:40.504 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:28:40.505 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:28:40.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:28:40.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:28:40.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:28:40.508 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:28:40.510 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:28:40.512 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:28:40.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:28:40.527 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:28:40.527 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:28:40.527 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:28:40.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:28:40.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:28:40.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:28:40.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:28:40.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_pg td/ceph-helpers 2023-09-30T01:28:40.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: local dir=td/ceph-helpers 2023-09-30T01:28:40.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: test_get_pg: setup td/ceph-helpers 2023-09-30T01:28:40.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:28:40.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:28:40.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:28:40.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:28:40.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:28:40.531 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:28:40.531 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:28:40.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:28:40.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:28:40.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:28:40.534 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:28:40.534 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:28:40.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:28:40.536 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:28:40.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:28:40.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:28:40.538 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:28:40.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:28:40.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:28:40.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:28:40.541 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:28:40.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:28:40.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:28:40.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:28:40.546 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:28:40.546 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:28:40.546 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:28:40.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:28:40.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:28:40.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:28:40.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:28:40.550 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:28:40.550 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:28:40.550 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:28:40.551 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:28:40.553 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:28:40.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:28:40.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:28:40.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:28:40.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:28:40.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:28:40.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:28:40.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:28:40.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:28:40.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:28:40.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:28:40.971 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:28:40.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:28:40.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:28:40.972 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:28:40.973 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:28:40.973 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:28:40.973 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:28:40.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:28:41.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:28:41.262 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:28:41.262 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:28:41.262 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:28:41.263 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:28:41.263 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:28:41.264 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:28:41.264 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:28:41.264 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:28:41.268 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:28:41.269 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:28:41.269 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:28:41.269 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:28:41.270 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:28:41.270 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:28:41.361 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:28:41.361 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:28:41.361 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:28:41.361 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:28:41.362 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:28:41.362 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:28:41.362 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:28:41.362 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:28:41.363 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:28:41.363 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:28:41.363 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:28:41.363 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:28:41.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:28:41.364 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:28:41.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: test_get_pg: run_mgr td/ceph-helpers x 2023-09-30T01:28:41.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:28:41.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:28:41.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:28:41.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:28:41.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:28:41.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:28:42.347 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:28:42.348 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:28:42.348 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:28:42.348 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:28:42.349 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:28:42.349 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:28:42.349 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:28:42.350 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:28:42.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:28:42.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: test_get_pg: run_osd td/ceph-helpers 0 2023-09-30T01:28:42.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:28:42.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:28:42.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:28:42.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:28:42.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:28:42.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:28:42.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:28:42.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:28:42.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:28:42.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:28:42.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:28:42.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:28:42.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:28:42.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:28:42.386 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:28:42.386 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:28:42.386 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:28:42.388 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:28:42.388 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:28:42.388 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:28:42.388 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:28:42.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:28:42.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:28:42.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:28:42.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:28:42.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:28:42.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:28:42.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:28:42.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:28:42.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:28:42.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:28:42.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:28:42.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:28:42.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:28:42.390 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:28:42.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff2794c6-6824-4723-9932-90600dfbe046 2023-09-30T01:28:42.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ff2794c6-6824-4723-9932-90600dfbe046' 2023-09-30T01:28:42.391 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 ff2794c6-6824-4723-9932-90600dfbe046 2023-09-30T01:28:42.391 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:28:42.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKeRdlnQtsGBAAWIY17x2d0A2FY2s3uSOkUg== 2023-09-30T01:28:42.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKeRdlnQtsGBAAWIY17x2d0A2FY2s3uSOkUg=="}' 2023-09-30T01:28:42.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff2794c6-6824-4723-9932-90600dfbe046 -i td/ceph-helpers/0/new.json 2023-09-30T01:28:42.788 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:28:42.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:28:42.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKeRdlnQtsGBAAWIY17x2d0A2FY2s3uSOkUg== --osd-uuid ff2794c6-6824-4723-9932-90600dfbe046 2023-09-30T01:28:42.836 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:42.828+0000 7f77e9b173c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:28:42.839 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:42.832+0000 7f77e9b173c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:28:42.841 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:42.836+0000 7f77e9b173c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:28:42.842 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:42.836+0000 7f77e9b173c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:28:42.842 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:42.836+0000 7f77e9b173c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:28:42.842 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:42.836+0000 7f77e9b173c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:28:42.919 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:42.912+0000 7f77e9b173c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:28:49.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:28:49.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:28:49.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:28:49.324 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:28:49.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:28:49.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:28:49.890 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:28:49.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:28:49.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:28:49.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:28:49.891 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:28:49.919 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:49.912+0000 7f3b99bf63c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:28:49.922 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:49.916+0000 7f3b99bf63c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:28:49.924 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:49.916+0000 7f3b99bf63c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:28:50.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:28:50.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:28:50.410 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:28:50.410 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:28:50.410 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:28:50.410 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:50.410 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:28:50.410 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:28:50.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:50.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:28:50.494 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:50.488+0000 7f3b99bf63c0 -1 Falling back to public interface 2023-09-30T01:28:50.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:51.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:51.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:51.854 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:28:51.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:28:51.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:51.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:28:52.273 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:53.062 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:28:53.056+0000 7f3b99bf63c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:28:53.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:53.275 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:53.275 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:28:53.275 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:28:53.275 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:53.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:28:53.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:54.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:54.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:54.889 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:28:54.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:28:54.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:54.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:28:55.294 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:56.295 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:56.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:56.296 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:28:56.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:28:56.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:56.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:28:56.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:57.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:57.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:57.709 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:28:57.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:28:57.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:57.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:28:58.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:28:59.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:28:59.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:28:59.136 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:28:59.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:28:59.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:28:59.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:28:59.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:00.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:00.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:00.549 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:29:00.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:29:00.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:00.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:00.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:01.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:01.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:01.962 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:29:01.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:29:01.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:01.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:02.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:03.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:03.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:03.384 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:29:03.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:29:03.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:03.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:03.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:04.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:04.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:04.819 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:29:04.820 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:29:04.820 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:04.820 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:05.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:06.568 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:06.568 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:06.568 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:29:06.569 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:29:06.569 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:06.569 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:07.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:08.220 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:08.220 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:08.221 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:29:08.221 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:29:08.221 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:08.221 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:09.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:10.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:10.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:10.095 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:29:10.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:29:10.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:10.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:10.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:11.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:11.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:11.681 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:29:11.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:29:11.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:11.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:12.458 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:13.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:13.461 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:13.461 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:29:13.461 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:29:13.461 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:13.462 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:14.420 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:15.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:15.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:15.423 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:29:15.423 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:29:15.423 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:15.423 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:15.833 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/116908,v1:127.0.0.1:6803/116908] [v2:127.0.0.1:6804/116908,v1:127.0.0.1:6805/116908] exists,up ff2794c6-6824-4723-9932-90600dfbe046 2023-09-30T01:29:15.834 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:29:15.834 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:29:15.834 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:29:15.834 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: test_get_pg: create_rbd_pool 2023-09-30T01:29:15.835 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:29:16.256 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:29:16.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:29:16.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:29:16.819 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:29:16.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:29:17.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:29:18.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: test_get_pg: wait_for_clean 2023-09-30T01:29:18.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:29:18.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:29:18.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:29:18.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:29:18.869 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:29:18.870 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:29:18.870 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:29:18.870 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:29:18.870 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:29:18.871 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:29:19.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:29:19.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:29:19.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:29:19.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:29:19.075 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:29:19.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:29:19.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:29:19.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:29:19.530 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:29:19.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836486 2023-09-30T01:29:19.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836486 2023-09-30T01:29:19.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836486' 2023-09-30T01:29:19.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:29:19.746 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836486 2023-09-30T01:29:19.746 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:29:19.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:29:19.749 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:29:19.749 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836486 2023-09-30T01:29:19.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836486 2023-09-30T01:29:19.751 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836486 2023-09-30T01:29:19.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2023-09-30T01:29:19.751 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:29:20.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836484 -lt 21474836486 2023-09-30T01:29:20.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:29:21.160 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:29:21.160 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:29:21.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836484 -lt 21474836486 2023-09-30T01:29:21.565 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:29:22.567 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 299 -eq 0 ']' 2023-09-30T01:29:22.567 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:29:22.976 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836487 -lt 21474836486 2023-09-30T01:29:22.977 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:29:22.977 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:29:22.977 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:29:23.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:29:23.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:29:23.524 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:29:23.524 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:29:23.524 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:29:23.524 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:29:23.524 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:29:23.525 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:29:23.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:29:23.886 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:29:23.887 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:29:23.887 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:29:24.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:29:24.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:29:24.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:29:24.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: get_pg rbd GROUP 2023-09-30T01:29:24.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2023-09-30T01:29:24.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=GROUP 2023-09-30T01:29:24.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2023-09-30T01:29:24.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd GROUP 2023-09-30T01:29:24.393 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2023-09-30T01:29:24.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1109: test_get_pg: teardown td/ceph-helpers 2023-09-30T01:29:24.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:29:24.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:29:24.818 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:29:24.818 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:29:24.818 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:29:24.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:29:24.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:29:24.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:29:24.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:29:24.931 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:29:24.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:29:24.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:29:24.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:29:24.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:29:24.936 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:29:24.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:29:24.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:29:24.939 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:29:24.940 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:29:24.940 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:29:24.942 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:29:24.942 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:29:24.952 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:29:24.952 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:29:24.952 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:29:24.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:29:24.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:29:24.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:29:24.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:29:24.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_primary td/ceph-helpers 2023-09-30T01:29:24.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: local dir=td/ceph-helpers 2023-09-30T01:29:24.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: test_get_primary: setup td/ceph-helpers 2023-09-30T01:29:24.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:29:24.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:29:24.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:29:24.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:29:24.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:29:24.956 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:29:24.956 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:29:24.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:29:24.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:29:24.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:29:24.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:29:24.960 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:29:24.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:29:24.962 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:29:24.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:29:24.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:29:24.965 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:29:24.966 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:29:24.966 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:29:24.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:29:24.967 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:29:24.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:29:24.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:29:24.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:29:24.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:29:24.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:29:24.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:29:24.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:29:24.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:29:24.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:29:24.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:29:24.976 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:29:24.976 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:29:24.976 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:29:24.976 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:29:24.978 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:29:24.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:29:24.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:29:24.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:29:24.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:29:24.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:29:24.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:29:24.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:29:24.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:29:24.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:29:24.980 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:29:25.598 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:29:25.598 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:29:25.598 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:29:25.599 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:29:25.599 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:29:25.599 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:29:25.600 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:29:25.600 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:29:25.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:29:25.889 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:29:25.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:29:25.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:29:25.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:29:25.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:29:25.891 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:29:25.891 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:29:25.891 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:29:25.893 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:29:25.893 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:29:25.893 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:29:25.894 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:29:25.894 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:29:25.894 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:29:25.984 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:29:25.984 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:29:25.985 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:29:25.985 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:29:25.986 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:29:25.986 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:29:25.986 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:29:25.986 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:29:25.986 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:29:25.987 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:29:25.987 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:29:25.987 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:29:25.987 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:29:25.987 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:29:26.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: test_get_primary: local osd=0 2023-09-30T01:29:26.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: test_get_primary: run_mgr td/ceph-helpers x 2023-09-30T01:29:26.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:29:26.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:29:26.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:29:26.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:29:26.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:29:26.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:29:26.970 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:29:26.970 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:29:26.970 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:29:26.970 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:29:26.970 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:29:26.971 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:29:26.971 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:29:26.971 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:29:26.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:29:27.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: test_get_primary: run_osd td/ceph-helpers 0 2023-09-30T01:29:27.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:29:27.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:29:27.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:29:27.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:29:27.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:29:27.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:29:27.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:29:27.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:29:27.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:29:27.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:29:27.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:29:27.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:29:27.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:29:27.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:29:27.010 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:29:27.010 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:29:27.010 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:29:27.011 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:29:27.011 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:29:27.011 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:29:27.011 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:29:27.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:29:27.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:29:27.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:29:27.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:29:27.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:29:27.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:29:27.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:29:27.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:29:27.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:29:27.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:29:27.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:29:27.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:29:27.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:29:27.013 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:29:27.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=71eb3c83-51a0-4c8a-bc6f-be43ca31074e 2023-09-30T01:29:27.014 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 71eb3c83-51a0-4c8a-bc6f-be43ca31074e 2023-09-30T01:29:27.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 71eb3c83-51a0-4c8a-bc6f-be43ca31074e' 2023-09-30T01:29:27.014 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:29:27.035 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3eRdlp+YKAhAAkZ6HOFVa3y56wsT3w0vJPw== 2023-09-30T01:29:27.035 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3eRdlp+YKAhAAkZ6HOFVa3y56wsT3w0vJPw=="}' 2023-09-30T01:29:27.035 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 71eb3c83-51a0-4c8a-bc6f-be43ca31074e -i td/ceph-helpers/0/new.json 2023-09-30T01:29:27.400 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:29:27.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:29:27.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3eRdlp+YKAhAAkZ6HOFVa3y56wsT3w0vJPw== --osd-uuid 71eb3c83-51a0-4c8a-bc6f-be43ca31074e 2023-09-30T01:29:27.447 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:27.443+0000 7f02d26e03c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:29:27.450 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:27.447+0000 7f02d26e03c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:29:27.451 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:27.447+0000 7f02d26e03c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:29:27.452 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:27.451+0000 7f02d26e03c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:29:27.453 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:27.451+0000 7f02d26e03c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:29:27.453 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:27.451+0000 7f02d26e03c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:29:27.523 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:27.519+0000 7f02d26e03c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:29:33.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:29:33.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:29:33.860 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:29:33.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:29:33.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:29:34.448 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:29:34.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:29:34.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:29:34.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:29:34.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:29:34.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:29:34.477 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:34.475+0000 7fb2b1f8f3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:29:34.494 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:34.491+0000 7fb2b1f8f3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:29:34.495 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:34.491+0000 7fb2b1f8f3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:29:34.847 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:29:34.848 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:29:34.848 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:29:34.848 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:29:34.848 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:29:34.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:29:34.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:34.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:29:34.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:34.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:35.046 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:35.043+0000 7fb2b1f8f3c0 -1 Falling back to public interface 2023-09-30T01:29:35.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:36.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:36.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:36.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:29:36.231 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:29:36.232 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:36.232 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:36.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:37.445 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:29:37.439+0000 7fb2b1f8f3c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:29:37.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:37.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:37.688 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:29:37.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:29:37.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:37.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:38.185 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:39.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:39.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:39.187 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:29:39.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:29:39.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:39.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:39.595 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:40.597 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:40.597 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:40.597 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:29:40.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:29:40.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:40.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:41.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:42.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:42.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:42.190 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:29:42.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:29:42.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:42.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:42.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:43.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:43.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:43.644 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:29:43.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:29:43.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:43.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:44.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:45.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:45.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:45.131 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:29:45.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:29:45.132 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:45.132 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:45.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:46.551 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:46.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:46.552 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:29:46.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:29:46.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:46.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:46.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:47.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:47.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:47.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:29:47.958 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:29:47.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:47.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:48.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:49.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:49.540 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:49.540 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:29:49.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:29:49.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:49.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:50.005 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:51.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:51.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:51.008 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:29:51.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:29:51.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:51.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:51.584 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:52.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:52.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:52.586 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:29:52.586 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:29:52.586 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:52.586 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:53.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:54.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:54.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:54.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:29:54.620 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:29:54.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:54.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:55.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:56.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:56.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:56.167 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:29:56.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:29:56.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:56.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:56.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:29:57.936 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:29:57.936 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:29:57.936 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:29:57.936 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:29:57.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:29:57.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:29:59.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:00.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:00.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:00.033 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:30:00.034 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:30:00.034 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:00.034 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:00.492 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/119356,v1:127.0.0.1:6803/119356] [v2:127.0.0.1:6804/119356,v1:127.0.0.1:6805/119356] exists,up 71eb3c83-51a0-4c8a-bc6f-be43ca31074e 2023-09-30T01:30:00.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:30:00.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:30:00.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:30:00.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: test_get_primary: create_rbd_pool 2023-09-30T01:30:00.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:30:00.899 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:30:00.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:30:00.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:30:01.533 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:30:01.544 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:30:02.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:30:03.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: test_get_primary: wait_for_clean 2023-09-30T01:30:03.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:30:03.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:30:03.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:30:03.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:30:03.590 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:30:03.591 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:03.591 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:03.591 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:03.591 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:03.591 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:03.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:30:03.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:30:03.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:30:03.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:30:03.801 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:30:04.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:30:04.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:30:04.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:30:04.246 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:30:04.423 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836490 2023-09-30T01:30:04.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836490 2023-09-30T01:30:04.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836490' 2023-09-30T01:30:04.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:30:04.424 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:30:04.424 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836490 2023-09-30T01:30:04.426 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:30:04.426 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:30:04.427 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836490 2023-09-30T01:30:04.427 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836490 2023-09-30T01:30:04.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2023-09-30T01:30:04.428 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836490 2023-09-30T01:30:04.428 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:30:04.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836488 -lt 21474836490 2023-09-30T01:30:04.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:30:05.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:30:05.819 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:30:06.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836491 -lt 21474836490 2023-09-30T01:30:06.242 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:30:06.242 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:30:06.243 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:30:06.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:30:06.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:30:06.814 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:30:06.814 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:30:06.814 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:30:06.815 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:30:06.815 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:30:06.815 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:30:07.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:30:07.207 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:30:07.207 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:30:07.207 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:30:07.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:30:07.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:30:07.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:30:07.707 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: get_primary rbd GROUP 2023-09-30T01:30:07.707 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2023-09-30T01:30:07.707 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2023-09-30T01:30:07.708 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2023-09-30T01:30:07.708 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2023-09-30T01:30:08.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: test 0 = 0 2023-09-30T01:30:08.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: test_get_primary: teardown td/ceph-helpers 2023-09-30T01:30:08.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:30:08.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:30:08.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:30:08.203 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:30:08.205 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:30:08.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:30:08.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:30:08.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:30:08.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:30:08.317 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:30:08.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:30:08.319 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:30:08.321 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:30:08.321 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:30:08.322 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:30:08.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:30:08.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:30:08.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:30:08.325 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:30:08.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:30:08.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:30:08.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:30:08.339 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:30:08.339 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:08.339 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:08.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:30:08.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:30:08.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:30:08.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:30:08.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_get_timeout_delays td/ceph-helpers 2023-09-30T01:30:08.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 1 2023-09-30T01:30:08.342 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:08.342 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:08.343 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:08.343 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:08.343 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:08.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '1 ' = '1 ' 2023-09-30T01:30:08.367 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 5 2023-09-30T01:30:08.369 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:08.369 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:08.369 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:08.369 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:08.369 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:08.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2023-09-30T01:30:08.416 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 6 2023-09-30T01:30:08.418 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:08.418 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:08.418 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:08.418 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:08.419 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:08.466 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2023-09-30T01:30:08.467 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 7 2023-09-30T01:30:08.468 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:08.468 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:08.469 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:08.469 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:08.469 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:08.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2023-09-30T01:30:08.528 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: get_timeout_delays 8 2023-09-30T01:30:08.529 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:08.529 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:08.529 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:08.530 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:08.530 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:08.591 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2023-09-30T01:30:08.592 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 .1 2023-09-30T01:30:08.593 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:08.593 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:08.594 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:08.594 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:08.594 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:08.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2023-09-30T01:30:08.660 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 1.5 .1 2023-09-30T01:30:08.661 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:08.661 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:08.662 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:08.662 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:08.662 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:08.741 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2023-09-30T01:30:08.742 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 5 .1 2023-09-30T01:30:08.743 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:08.743 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:08.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:08.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:08.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:08.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2023-09-30T01:30:08.841 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 6 .1 2023-09-30T01:30:08.842 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:08.842 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:08.843 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:08.843 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:08.843 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:08.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2023-09-30T01:30:08.942 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 6.3 .1 2023-09-30T01:30:08.942 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:08.942 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:08.943 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:08.943 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:08.943 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:09.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2023-09-30T01:30:09.041 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 20 .1 2023-09-30T01:30:09.041 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:09.042 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:09.042 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:09.042 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:09.042 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:09.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2023-09-30T01:30:09.175 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 300 .1 0 2023-09-30T01:30:09.176 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:09.176 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:09.177 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:09.177 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:09.177 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:09.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2023-09-30T01:30:09.384 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 300 .1 10 2023-09-30T01:30:09.385 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:09.385 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:09.386 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:09.386 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:09.386 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:09.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2023-09-30T01:30:09.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:30:09.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_is_clean td/ceph-helpers 2023-09-30T01:30:09.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: local dir=td/ceph-helpers 2023-09-30T01:30:09.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: test_is_clean: setup td/ceph-helpers 2023-09-30T01:30:09.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:30:09.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:30:09.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:30:09.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:30:09.992 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:30:09.993 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:30:09.993 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:30:09.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:30:09.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:30:09.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:30:09.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:30:09.995 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:30:09.996 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:30:09.997 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:30:09.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:30:09.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:30:10.000 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:30:10.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:30:10.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:30:10.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:30:10.003 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:30:10.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:30:10.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:30:10.007 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:30:10.009 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:30:10.009 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:10.009 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:10.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:30:10.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:30:10.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:30:10.011 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:30:10.014 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:30:10.014 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:10.014 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:10.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:30:10.017 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:30:10.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:30:10.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:30:10.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:30:10.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:30:10.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:30:10.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:30:10.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:30:10.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:30:10.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:30:10.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:30:10.428 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:30:10.428 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:30:10.429 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:30:10.429 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:30:10.429 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:10.429 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:10.430 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:30:10.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:30:10.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:30:10.719 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:30:10.719 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:30:10.719 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:30:10.720 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:30:10.720 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:30:10.720 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:30:10.721 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:30:10.721 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:30:10.721 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:30:10.721 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:10.722 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:10.722 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:30:10.722 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:30:10.722 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:30:10.811 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:30:10.811 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:30:10.811 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:30:10.811 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:30:10.812 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:30:10.812 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:30:10.812 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:30:10.812 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:30:10.813 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:30:10.813 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:10.813 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:10.813 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:30:10.814 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:30:10.814 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:30:10.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: test_is_clean: run_mgr td/ceph-helpers x 2023-09-30T01:30:10.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:30:10.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:30:10.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:30:10.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:30:10.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:30:10.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:30:11.800 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:30:11.800 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:30:11.800 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:30:11.801 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:30:11.801 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:11.801 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:11.801 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:30:11.802 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:30:11.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:30:11.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: test_is_clean: run_osd td/ceph-helpers 0 2023-09-30T01:30:11.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:30:11.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:30:11.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:30:11.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:30:11.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:30:11.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:30:11.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:30:11.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:30:11.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:30:11.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:30:11.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:30:11.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:30:11.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:30:11.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:30:11.843 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:30:11.843 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:30:11.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:30:11.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:30:11.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:30:11.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:30:11.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:30:11.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:30:11.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:30:11.846 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:30:11.846 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c20e068-9fef-4c99-925f-feb4ed772b85 2023-09-30T01:30:11.846 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5c20e068-9fef-4c99-925f-feb4ed772b85' 2023-09-30T01:30:11.846 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 5c20e068-9fef-4c99-925f-feb4ed772b85 2023-09-30T01:30:11.846 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:30:11.866 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAjehdlSdiPMxAAy4Ico7KaA7PxSzO+OnOwSQ== 2023-09-30T01:30:11.866 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAjehdlSdiPMxAAy4Ico7KaA7PxSzO+OnOwSQ=="}' 2023-09-30T01:30:11.866 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c20e068-9fef-4c99-925f-feb4ed772b85 -i td/ceph-helpers/0/new.json 2023-09-30T01:30:12.229 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:30:12.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:30:12.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAjehdlSdiPMxAAy4Ico7KaA7PxSzO+OnOwSQ== --osd-uuid 5c20e068-9fef-4c99-925f-feb4ed772b85 2023-09-30T01:30:12.272 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:12.271+0000 7f58850c53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:30:12.275 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:12.271+0000 7f58850c53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:30:12.277 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:12.275+0000 7f58850c53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:30:12.278 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:12.275+0000 7f58850c53c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:30:12.278 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:12.275+0000 7f58850c53c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:30:12.279 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:12.275+0000 7f58850c53c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:30:12.359 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:12.355+0000 7f58850c53c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:30:18.741 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:30:18.742 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:30:18.743 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:30:18.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:30:18.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:30:19.340 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:30:19.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:30:19.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:30:19.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:30:19.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:30:19.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:30:19.373 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:19.371+0000 7f5d8b3a33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:30:19.384 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:19.383+0000 7f5d8b3a33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:30:19.386 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:19.383+0000 7f5d8b3a33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:30:19.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:30:19.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:30:19.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:30:19.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:30:19.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:30:19.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:19.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:30:19.748 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:30:19.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:19.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:19.934 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:19.931+0000 7f5d8b3a33c0 -1 Falling back to public interface 2023-09-30T01:30:20.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:21.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:21.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:21.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:30:21.135 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:30:21.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:21.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:21.534 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:22.189 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:22.187+0000 7f5d8b3a33c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:30:22.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:22.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:22.537 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:30:22.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:30:22.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:22.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:23.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:24.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:24.203 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:24.203 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:30:24.204 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:30:24.204 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:24.204 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:24.636 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:25.637 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:25.637 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:25.638 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:30:25.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:30:25.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:25.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:26.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:27.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:27.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:27.290 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:30:27.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:30:27.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:27.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:27.714 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:28.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:28.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:28.716 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:30:28.717 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:30:28.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:28.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:29.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:30.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:30.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:30.176 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:30:30.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:30:30.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:30.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:30.593 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:31.597 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:31.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:31.598 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:30:31.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:30:31.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:31.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:32.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:33.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:33.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:33.043 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:30:33.043 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:30:33.043 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:33.043 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:33.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:34.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:34.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:34.451 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:30:34.452 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:30:34.452 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:34.452 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:35.023 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:36.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:36.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:36.025 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:30:36.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:30:36.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:36.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:36.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:37.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:37.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:37.973 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:30:37.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:30:37.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:37.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:38.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:39.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:39.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:39.779 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:30:39.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:30:39.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:39.780 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:40.471 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:41.474 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:41.474 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:41.474 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:30:41.474 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:30:41.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:41.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:42.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:43.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:43.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:43.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:30:43.253 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:30:43.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:43.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:44.034 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:30:45.036 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:30:45.036 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:30:45.036 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:30:45.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:30:45.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:30:45.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:30:45.555 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/122565,v1:127.0.0.1:6803/122565] [v2:127.0.0.1:6804/122565,v1:127.0.0.1:6805/122565] exists,up 5c20e068-9fef-4c99-925f-feb4ed772b85 2023-09-30T01:30:45.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:30:45.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:30:45.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:30:45.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: test_is_clean: create_rbd_pool 2023-09-30T01:30:45.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:30:45.998 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:30:46.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:30:46.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:30:46.755 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:30:46.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:30:47.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:30:48.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: test_is_clean: wait_for_clean 2023-09-30T01:30:48.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:30:48.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:30:48.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:30:48.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:30:48.906 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:30:48.906 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:30:48.906 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:30:48.907 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:30:48.907 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:30:48.907 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:30:49.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:30:49.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:30:49.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:30:49.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:30:49.115 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:30:49.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:30:49.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:30:49.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:30:49.543 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:30:49.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836490 2023-09-30T01:30:49.683 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836490 2023-09-30T01:30:49.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836490' 2023-09-30T01:30:49.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:30:49.684 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836490 2023-09-30T01:30:49.684 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:30:49.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:30:49.687 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836490 2023-09-30T01:30:49.687 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:30:49.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836490 2023-09-30T01:30:49.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2023-09-30T01:30:49.689 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836490 2023-09-30T01:30:49.689 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:30:50.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836488 -lt 21474836490 2023-09-30T01:30:50.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:30:51.127 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:30:51.128 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:30:51.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836491 -lt 21474836490 2023-09-30T01:30:51.655 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:30:51.656 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:30:51.656 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:30:52.219 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:30:52.219 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:30:52.220 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:30:52.220 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:30:52.220 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:30:52.220 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:30:52.220 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:30:52.221 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:30:52.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:30:52.643 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:30:52.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:30:52.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:30:53.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:30:53.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:30:53.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:30:53.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1569: test_is_clean: is_clean 2023-09-30T01:30:53.174 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: get_num_pgs 2023-09-30T01:30:53.174 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:30:53.174 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:30:53.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: num_pgs=4 2023-09-30T01:30:53.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: is_clean: test 4 '!=' 0 2023-09-30T01:30:53.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: get_num_active_clean 2023-09-30T01:30:53.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:30:53.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:30:53.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:30:53.737 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:30:53.737 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:30:54.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: test 4 = 4 2023-09-30T01:30:54.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: test_is_clean: teardown td/ceph-helpers 2023-09-30T01:30:54.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:30:54.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:30:54.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:30:54.093 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:30:54.093 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:30:54.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:30:54.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:30:54.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:30:54.207 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:30:54.208 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:30:54.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:30:54.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:30:54.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:30:54.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:30:54.213 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:30:54.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:30:54.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:30:54.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:30:54.216 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:30:54.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:30:54.220 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:30:54.220 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:30:54.231 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:30:54.231 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:54.231 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:54.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:30:54.233 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:30:54.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:30:54.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:30:54.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_kill_daemon td/ceph-helpers 2023-09-30T01:30:54.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:275: test_kill_daemon: local dir=td/ceph-helpers 2023-09-30T01:30:54.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:276: test_kill_daemon: setup td/ceph-helpers 2023-09-30T01:30:54.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:30:54.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:30:54.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:30:54.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:30:54.234 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:30:54.235 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:30:54.235 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:30:54.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:30:54.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:30:54.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:30:54.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:30:54.240 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:30:54.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:30:54.242 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:30:54.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:30:54.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:30:54.244 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:30:54.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:30:54.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:30:54.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:30:54.247 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:30:54.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:30:54.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:30:54.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:30:54.252 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:30:54.253 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:54.253 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:54.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:30:54.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:30:54.255 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:30:54.255 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:30:54.257 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:30:54.257 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:54.257 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:54.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:30:54.260 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:30:54.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:30:54.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:30:54.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:30:54.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:277: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:30:54.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:30:54.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:30:54.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:30:54.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:30:54.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:30:54.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:30:54.632 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:30:54.633 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:30:54.633 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:30:54.633 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:30:54.634 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:54.634 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:54.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:30:54.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:30:54.913 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:30:54.915 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:30:54.915 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:30:54.915 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:30:54.915 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:30:54.916 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:30:54.916 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:30:54.916 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:30:54.916 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:30:54.917 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:30:54.917 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:54.917 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:54.918 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:30:54.918 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:30:54.918 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:30:55.014 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:30:55.014 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:30:55.014 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:30:55.015 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:30:55.015 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:30:55.016 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:30:55.016 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:30:55.016 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:30:55.016 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:30:55.016 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:55.016 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:55.017 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:30:55.017 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:30:55.017 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:30:55.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:278: test_kill_daemon: run_mgr td/ceph-helpers x 2023-09-30T01:30:55.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:30:55.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:30:55.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:30:55.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:30:55.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:30:55.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:30:56.029 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:30:56.029 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:30:56.029 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:30:56.029 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:30:56.030 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:56.030 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:56.030 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:30:56.030 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:30:56.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:30:56.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:279: test_kill_daemon: run_osd td/ceph-helpers 0 2023-09-30T01:30:56.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:30:56.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:30:56.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:30:56.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:30:56.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:30:56.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:30:56.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:30:56.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:30:56.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:30:56.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:30:56.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:30:56.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:30:56.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:30:56.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:30:56.064 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:30:56.064 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:30:56.064 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:30:56.065 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:30:56.065 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:30:56.065 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:30:56.065 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:30:56.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:30:56.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:30:56.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:30:56.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:30:56.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:30:56.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:30:56.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:30:56.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:30:56.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:30:56.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:30:56.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:30:56.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:30:56.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:30:56.066 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:30:56.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7751f9db-bc35-45bd-88bc-662286921718 2023-09-30T01:30:56.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7751f9db-bc35-45bd-88bc-662286921718' 2023-09-30T01:30:56.068 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 7751f9db-bc35-45bd-88bc-662286921718 2023-09-30T01:30:56.068 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:30:56.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQehdlKrAhBRAAQu86/+uQqLZayFp7ZuLLlg== 2023-09-30T01:30:56.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQehdlKrAhBRAAQu86/+uQqLZayFp7ZuLLlg=="}' 2023-09-30T01:30:56.087 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7751f9db-bc35-45bd-88bc-662286921718 -i td/ceph-helpers/0/new.json 2023-09-30T01:30:56.440 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:30:56.454 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:30:56.455 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQehdlKrAhBRAAQu86/+uQqLZayFp7ZuLLlg== --osd-uuid 7751f9db-bc35-45bd-88bc-662286921718 2023-09-30T01:30:56.481 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:56.478+0000 7fbee30703c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:30:56.484 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:56.482+0000 7fbee30703c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:30:56.486 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:56.482+0000 7fbee30703c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:30:56.487 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:56.482+0000 7fbee30703c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:30:56.487 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:56.482+0000 7fbee30703c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:30:56.487 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:56.482+0000 7fbee30703c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:30:56.596 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:30:56.594+0000 7fbee30703c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:31:03.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:31:03.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:31:03.071 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:31:03.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:31:03.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:31:03.591 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:31:03.591 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:31:03.592 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:31:03.592 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:31:03.592 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:31:03.592 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:31:03.622 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:03.618+0000 7f29bbcee3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:31:03.625 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:03.622+0000 7f29bbcee3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:31:03.626 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:03.622+0000 7f29bbcee3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:31:04.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:31:04.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:31:04.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:31:04.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:31:04.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:31:04.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:04.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:31:04.032 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:31:04.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:04.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:04.198 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:04.194+0000 7f29bbcee3c0 -1 Falling back to public interface 2023-09-30T01:31:04.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:05.411 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:05.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:05.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:31:05.412 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:31:05.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:05.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:05.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:06.540 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:06.538+0000 7f29bbcee3c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:31:06.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:06.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:06.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:31:06.813 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:31:06.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:06.814 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:07.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:08.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:08.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:08.337 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:31:08.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:31:08.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:08.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:08.817 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:09.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:09.819 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:09.819 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:31:09.820 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:31:09.820 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:09.820 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:10.356 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:11.357 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:11.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:11.358 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:31:11.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:31:11.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:11.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:11.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:12.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:12.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:12.771 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:31:12.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:31:12.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:12.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:13.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:14.269 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:14.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:14.270 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:31:14.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:31:14.271 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:14.271 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:14.677 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:15.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:15.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:15.680 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:31:15.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:31:15.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:15.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:16.089 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:17.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:17.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:17.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:31:17.091 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:31:17.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:17.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:17.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:18.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:18.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:18.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:31:18.526 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:31:18.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:18.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:19.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:20.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:20.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:20.064 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:31:20.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:31:20.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:20.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:21.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:22.020 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:22.021 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:22.021 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:31:22.022 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:31:22.022 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:22.022 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:22.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:23.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:23.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:23.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:31:23.954 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:31:23.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:23.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:24.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:25.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:25.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:25.538 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:31:25.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:31:25.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:25.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:26.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:27.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:27.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:27.009 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:31:27.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:31:27.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:27.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:27.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:28.760 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:28.760 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:28.760 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:31:28.761 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:31:28.761 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:28.761 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:29.194 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/125013,v1:127.0.0.1:6803/125013] [v2:127.0.0.1:6804/125013,v1:127.0.0.1:6805/125013] exists,up 7751f9db-bc35-45bd-88bc-662286921718 2023-09-30T01:31:29.194 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:31:29.195 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:31:29.195 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:31:29.195 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:281: test_kill_daemon: name_prefix=osd 2023-09-30T01:31:29.195 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: find td/ceph-helpers 2023-09-30T01:31:29.195 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: grep 'osd[^/]*\.pid' 2023-09-30T01:31:29.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2023-09-30T01:31:29.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:289: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2023-09-30T01:31:29.198 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2023-09-30T01:31:29.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=125013 2023-09-30T01:31:29.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=0 2023-09-30T01:31:29.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local delays=1 2023-09-30T01:31:29.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2023-09-30T01:31:29.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2023-09-30T01:31:29.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 125013 2023-09-30T01:31:29.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2023-09-30T01:31:29.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2023-09-30T01:31:29.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2023-09-30T01:31:30.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2023-09-30T01:31:30.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 125013 2023-09-30T01:31:30.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2023-09-30T01:31:30.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2023-09-30T01:31:30.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0 2023-09-30T01:31:30.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 1 2023-09-30T01:31:30.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2023-09-30T01:31:30.203 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2023-09-30T01:31:30.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=125013 2023-09-30T01:31:30.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2023-09-30T01:31:30.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2023-09-30T01:31:30.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2023-09-30T01:31:30.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2023-09-30T01:31:30.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 125013 2023-09-30T01:31:30.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2023-09-30T01:31:30.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2023-09-30T01:31:30.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2023-09-30T01:31:30.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2023-09-30T01:31:30.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 125013 2023-09-30T01:31:30.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2023-09-30T01:31:30.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2023-09-30T01:31:30.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2023-09-30T01:31:30.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2023-09-30T01:31:30.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 125013 2023-09-30T01:31:30.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2023-09-30T01:31:30.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2023-09-30T01:31:30.509 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2023-09-30T01:31:31.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2023-09-30T01:31:31.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 125013 2023-09-30T01:31:31.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2023-09-30T01:31:31.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2023-09-30T01:31:31.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2023-09-30T01:31:31.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: name_prefix=mgr 2023-09-30T01:31:31.512 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: find td/ceph-helpers 2023-09-30T01:31:31.512 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: grep 'mgr[^/]*\.pid' 2023-09-30T01:31:31.515 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2023-09-30T01:31:31.515 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2023-09-30T01:31:31.516 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2023-09-30T01:31:31.518 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=124033 2023-09-30T01:31:31.518 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2023-09-30T01:31:31.518 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2023-09-30T01:31:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2023-09-30T01:31:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2023-09-30T01:31:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 124033 2023-09-30T01:31:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2023-09-30T01:31:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2023-09-30T01:31:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2023-09-30T01:31:31.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2023-09-30T01:31:31.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 124033 2023-09-30T01:31:31.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2023-09-30T01:31:31.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2023-09-30T01:31:31.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2023-09-30T01:31:31.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: name_prefix=mon 2023-09-30T01:31:31.623 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: find td/ceph-helpers 2023-09-30T01:31:31.623 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: grep 'mon[^/]*\.pid' 2023-09-30T01:31:31.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2023-09-30T01:31:31.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:309: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2023-09-30T01:31:31.626 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mon.a.pid 2023-09-30T01:31:31.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=123945 2023-09-30T01:31:31.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2023-09-30T01:31:31.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2023-09-30T01:31:31.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2023-09-30T01:31:31.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2023-09-30T01:31:31.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 123945 2023-09-30T01:31:31.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2023-09-30T01:31:31.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2023-09-30T01:31:31.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2023-09-30T01:31:31.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2023-09-30T01:31:31.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 123945 2023-09-30T01:31:31.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2023-09-30T01:31:31.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2023-09-30T01:31:31.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2023-09-30T01:31:31.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:310: test_kill_daemon: timeout 5 ceph status 2023-09-30T01:31:36.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:313: test_kill_daemon: teardown td/ceph-helpers 2023-09-30T01:31:36.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:31:36.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:31:36.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:31:36.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:31:36.737 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:31:36.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:31:36.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:31:36.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:31:36.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:31:36.747 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:31:36.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:31:36.749 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:31:36.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:31:36.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:31:36.751 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:31:36.753 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:31:36.753 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:31:36.753 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:31:36.754 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:31:36.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:31:36.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:31:36.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:31:36.768 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:31:36.768 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:31:36.768 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:31:36.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:31:36.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:31:36.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:31:36.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:31:36.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_kill_daemons td/ceph-helpers 2023-09-30T01:31:36.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:366: test_kill_daemons: local dir=td/ceph-helpers 2023-09-30T01:31:36.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:367: test_kill_daemons: setup td/ceph-helpers 2023-09-30T01:31:36.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:31:36.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:31:36.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:31:36.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:31:36.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:31:36.771 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:31:36.771 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:31:36.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:31:36.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:31:36.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:31:36.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:31:36.777 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:31:36.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:31:36.778 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:31:36.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:31:36.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:31:36.780 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:31:36.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:31:36.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:31:36.782 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:31:36.782 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:31:36.783 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:31:36.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:31:36.785 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:31:36.787 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:31:36.787 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:31:36.787 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:31:36.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:31:36.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:31:36.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:31:36.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:31:36.791 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:31:36.791 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:31:36.791 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:31:36.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:31:36.794 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:31:36.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:31:36.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:31:36.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:31:36.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:31:36.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:31:36.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:31:36.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:31:36.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:31:36.796 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:31:36.796 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:31:37.179 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:31:37.179 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:31:37.179 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:31:37.180 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:31:37.180 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:31:37.180 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:31:37.180 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:31:37.181 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:31:37.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:31:37.504 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:31:37.504 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:31:37.505 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:31:37.505 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:31:37.505 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:31:37.506 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:31:37.506 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:31:37.506 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:31:37.506 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:31:37.507 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:31:37.507 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:31:37.507 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:31:37.507 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:31:37.508 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:31:37.612 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:31:37.613 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:31:37.613 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:31:37.613 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:31:37.613 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:31:37.614 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:31:37.614 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:31:37.614 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:31:37.614 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:31:37.615 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:31:37.615 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:31:37.615 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:31:37.615 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:31:37.615 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:31:37.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:369: test_kill_daemons: run_mgr td/ceph-helpers x 2023-09-30T01:31:37.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:31:37.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:31:37.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:31:37.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:31:37.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:31:37.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:31:38.604 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:31:38.604 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:31:38.604 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:31:38.605 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:31:38.605 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:31:38.605 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:31:38.605 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:31:38.606 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:31:38.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:31:38.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:370: test_kill_daemons: run_osd td/ceph-helpers 0 2023-09-30T01:31:38.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:31:38.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:31:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:31:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:31:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:31:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:31:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:31:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:31:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:31:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:31:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:31:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:31:38.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:31:38.642 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:31:38.642 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:31:38.642 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:31:38.642 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:31:38.642 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:31:38.642 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:31:38.642 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:31:38.644 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:31:38.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:31:38.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:31:38.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:31:38.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:31:38.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:31:38.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:31:38.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:31:38.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:31:38.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:31:38.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:31:38.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:31:38.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:31:38.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:31:38.645 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:31:38.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b7ef3603-a82f-4054-9fc2-ae8252c331d6 2023-09-30T01:31:38.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b7ef3603-a82f-4054-9fc2-ae8252c331d6' 2023-09-30T01:31:38.646 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 b7ef3603-a82f-4054-9fc2-ae8252c331d6 2023-09-30T01:31:38.646 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:31:38.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB6ehdlgySpJxAA/fmxqpmdLMrGtA50FEcD/Q== 2023-09-30T01:31:38.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB6ehdlgySpJxAA/fmxqpmdLMrGtA50FEcD/Q=="}' 2023-09-30T01:31:38.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b7ef3603-a82f-4054-9fc2-ae8252c331d6 -i td/ceph-helpers/0/new.json 2023-09-30T01:31:39.039 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:31:39.052 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:31:39.054 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6ehdlgySpJxAA/fmxqpmdLMrGtA50FEcD/Q== --osd-uuid b7ef3603-a82f-4054-9fc2-ae8252c331d6 2023-09-30T01:31:39.087 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:39.081+0000 7fb4841e73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:31:39.089 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:39.085+0000 7fb4841e73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:31:39.090 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:39.085+0000 7fb4841e73c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:31:39.091 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:39.085+0000 7fb4841e73c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:31:39.091 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:39.085+0000 7fb4841e73c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:31:39.091 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:39.085+0000 7fb4841e73c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:31:39.163 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:39.157+0000 7fb4841e73c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:31:45.501 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:31:45.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:31:45.503 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:31:45.504 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:31:45.504 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:31:46.036 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:31:46.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:31:46.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:31:46.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:31:46.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:31:46.038 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:31:46.069 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:46.065+0000 7f0c685793c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:31:46.108 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:46.101+0000 7f0c685793c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:31:46.110 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:46.105+0000 7f0c685793c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:31:46.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:31:46.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:31:46.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:31:46.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:31:46.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:31:46.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:46.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:31:46.483 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:31:46.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:46.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:46.678 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:46.673+0000 7f0c685793c0 -1 Falling back to public interface 2023-09-30T01:31:46.928 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:47.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:47.931 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:47.931 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:31:47.931 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:31:47.931 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:47.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:48.321 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:49.043 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:31:49.037+0000 7f0c685793c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:31:49.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:49.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:49.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:31:49.323 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:31:49.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:49.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:49.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:50.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:50.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:50.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:31:50.733 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:31:50.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:50.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:51.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:52.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:52.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:52.190 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:31:52.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:31:52.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:52.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:52.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:53.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:53.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:53.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:31:53.775 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:31:53.775 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:53.775 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:54.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:55.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:55.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:55.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:31:55.217 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:31:55.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:55.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:55.710 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:56.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:56.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:56.712 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:31:56.713 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:31:56.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:56.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:57.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:58.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:58.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:58.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:31:58.145 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:31:58.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:58.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:31:58.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:31:59.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:31:59.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:31:59.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:31:59.555 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:31:59.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:31:59.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:00.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:01.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:01.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:01.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:32:01.249 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:32:01.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:01.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:01.848 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:02.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:02.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:02.849 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:32:02.850 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:32:02.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:02.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:03.427 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:04.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:04.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:04.429 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:32:04.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:32:04.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:04.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:05.442 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:06.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:06.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:06.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:32:06.445 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:32:06.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:06.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:07.026 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:08.027 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:08.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:08.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:32:08.028 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:32:08.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:08.029 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:08.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:09.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:09.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:09.803 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:32:09.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:32:09.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:09.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:10.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:11.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:11.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:11.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:32:11.525 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:32:11.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:11.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:11.933 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/127085,v1:127.0.0.1:6803/127085] [v2:127.0.0.1:6804/127085,v1:127.0.0.1:6805/127085] exists,up b7ef3603-a82f-4054-9fc2-ae8252c331d6 2023-09-30T01:32:11.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:32:11.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:32:11.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:32:11.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:377: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2023-09-30T01:32:11.934 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:32:11.935 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:32:11.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:32:11.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:32:11.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:32:12.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 1 2023-09-30T01:32:12.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2023-09-30T01:32:12.947 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:32:12.947 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:32:12.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:32:12.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:32:12.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:32:13.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:32:13.263 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:385: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2023-09-30T01:32:13.263 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:32:13.263 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:32:13.263 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:32:13.264 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:32:13.264 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:32:13.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:32:13.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:389: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2023-09-30T01:32:13.376 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:32:13.376 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:32:13.376 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:32:13.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:32:13.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:32:13.489 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:32:13.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:390: test_kill_daemons: timeout 5 ceph status 2023-09-30T01:32:18.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:391: test_kill_daemons: teardown td/ceph-helpers 2023-09-30T01:32:18.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:32:18.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:32:18.495 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:32:18.496 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:32:18.496 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:32:18.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:32:18.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:32:18.496 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:32:18.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:32:18.507 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:32:18.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:32:18.509 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:32:18.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:32:18.511 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:32:18.512 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:32:18.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:32:18.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:32:18.514 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:32:18.515 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:32:18.516 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:32:18.518 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:32:18.518 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:32:18.526 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:32:18.526 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:32:18.526 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:32:18.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:32:18.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:32:18.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:32:18.529 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:32:18.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_objectstore_tool td/ceph-helpers 2023-09-30T01:32:18.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: local dir=td/ceph-helpers 2023-09-30T01:32:18.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: setup td/ceph-helpers 2023-09-30T01:32:18.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:32:18.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:32:18.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:32:18.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:32:18.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:32:18.531 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:32:18.531 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:32:18.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:32:18.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:32:18.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:32:18.534 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:32:18.535 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:32:18.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:32:18.537 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:32:18.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:32:18.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:32:18.540 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:32:18.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:32:18.541 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:32:18.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:32:18.542 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:32:18.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:32:18.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:32:18.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:32:18.549 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:32:18.549 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:32:18.549 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:32:18.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:32:18.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:32:18.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:32:18.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:32:18.553 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:32:18.553 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:32:18.553 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:32:18.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:32:18.556 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:32:18.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:32:18.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:32:18.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:32:18.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:32:18.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:32:18.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:32:18.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:32:18.558 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:32:18.558 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:32:18.558 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:32:18.925 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:32:18.925 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:32:18.925 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:32:18.926 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:32:18.926 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:32:18.927 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:32:18.927 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:32:18.927 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:32:19.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:32:19.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:32:19.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:32:19.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:32:19.249 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:32:19.249 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:32:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:32:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:32:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:32:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:32:19.250 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:32:19.251 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:32:19.251 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:32:19.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:32:19.251 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:32:19.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:32:19.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:32:19.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:32:19.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:32:19.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:32:19.343 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:32:19.343 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:32:19.343 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:32:19.344 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:32:19.344 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:32:19.344 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:32:19.344 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:32:19.344 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:32:19.345 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:32:19.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: local osd=0 2023-09-30T01:32:19.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: test_objectstore_tool: run_mgr td/ceph-helpers x 2023-09-30T01:32:19.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:32:19.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:32:19.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:32:19.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:32:19.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:32:19.432 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:32:20.339 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:32:20.339 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:32:20.339 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:32:20.340 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:32:20.340 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:32:20.340 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:32:20.341 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:32:20.341 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:32:20.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:32:20.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: test_objectstore_tool: run_osd td/ceph-helpers 0 2023-09-30T01:32:20.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:32:20.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:32:20.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:32:20.379 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:32:20.379 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:32:20.379 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:32:20.380 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:32:20.380 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:32:20.381 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:32:20.381 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:32:20.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:32:20.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:32:20.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:32:20.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:32:20.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:32:20.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:32:20.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:32:20.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:32:20.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:32:20.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:32:20.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:32:20.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:32:20.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:32:20.382 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:32:20.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f4004270-c8ac-46b0-99d8-6fb1012a2f42 2023-09-30T01:32:20.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f4004270-c8ac-46b0-99d8-6fb1012a2f42' 2023-09-30T01:32:20.383 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 f4004270-c8ac-46b0-99d8-6fb1012a2f42 2023-09-30T01:32:20.383 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:32:20.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCkehdlr230FxAAq5dC6U+9OT06rvkvutk7fg== 2023-09-30T01:32:20.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCkehdlr230FxAAq5dC6U+9OT06rvkvutk7fg=="}' 2023-09-30T01:32:20.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f4004270-c8ac-46b0-99d8-6fb1012a2f42 -i td/ceph-helpers/0/new.json 2023-09-30T01:32:20.757 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:32:20.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:32:20.775 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCkehdlr230FxAAq5dC6U+9OT06rvkvutk7fg== --osd-uuid f4004270-c8ac-46b0-99d8-6fb1012a2f42 2023-09-30T01:32:20.806 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:20.800+0000 7f25dbb7b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:32:20.809 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:20.804+0000 7f25dbb7b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:32:20.810 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:20.804+0000 7f25dbb7b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:32:20.811 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:20.804+0000 7f25dbb7b3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:32:20.812 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:20.804+0000 7f25dbb7b3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:32:20.812 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:20.804+0000 7f25dbb7b3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:32:20.913 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:20.908+0000 7f25dbb7b3c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:32:27.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:32:27.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:32:27.329 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:32:27.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:32:27.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:32:27.873 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:32:27.874 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:32:27.874 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:32:27.874 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:32:27.874 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:32:27.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:32:27.905 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:27.900+0000 7f8b904e23c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:32:27.921 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:27.916+0000 7f8b904e23c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:32:27.923 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:27.916+0000 7f8b904e23c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:32:28.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:32:28.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:32:28.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:32:28.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:32:28.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:32:28.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:28.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:32:28.283 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:32:28.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:28.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:28.495 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:28.488+0000 7f8b904e23c0 -1 Falling back to public interface 2023-09-30T01:32:28.706 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:29.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:29.708 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:29.708 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:32:29.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:32:29.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:29.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:30.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:30.980 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:32:30.972+0000 7f8b904e23c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:32:31.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:31.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:31.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:32:31.119 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:32:31.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:31.120 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:31.671 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:32.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:32.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:32.673 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:32:32.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:32:32.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:32.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:33.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:34.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:34.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:34.131 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:32:34.132 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:32:34.132 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:34.132 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:34.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:35.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:35.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:35.690 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:32:35.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:32:35.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:35.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:36.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:37.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:37.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:37.096 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:32:37.097 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:32:37.097 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:37.097 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:37.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:38.505 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:38.505 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:38.505 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:32:38.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:32:38.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:38.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:39.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:40.003 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:40.003 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:40.004 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:32:40.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:32:40.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:40.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:40.427 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:41.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:41.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:41.430 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:32:41.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:32:41.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:41.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:42.104 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:43.107 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:43.107 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:43.107 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:32:43.108 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:32:43.108 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:43.108 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:43.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:44.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:44.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:44.522 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:32:44.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:32:44.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:44.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:45.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:46.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:46.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:46.129 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:32:46.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:32:46.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:46.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:47.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:48.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:48.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:48.126 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:32:48.127 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:32:48.127 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:48.127 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:48.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:49.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:49.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:49.665 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:32:49.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:32:49.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:49.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:50.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:51.500 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:51.500 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:51.500 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:32:51.501 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:32:51.501 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:51.501 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:52.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:32:53.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:32:53.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:32:53.252 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:32:53.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:32:53.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:32:53.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:32:53.669 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/129179,v1:127.0.0.1:6803/129179] [v2:127.0.0.1:6804/129179,v1:127.0.0.1:6805/129179] exists,up f4004270-c8ac-46b0-99d8-6fb1012a2f42 2023-09-30T01:32:53.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:32:53.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:32:53.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:32:53.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: test_objectstore_tool: create_rbd_pool 2023-09-30T01:32:53.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:32:54.149 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:32:54.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:32:54.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:32:54.874 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:32:54.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:32:55.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:32:56.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: test_objectstore_tool: wait_for_clean 2023-09-30T01:32:56.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:32:56.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:32:56.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:32:56.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:32:56.988 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:32:56.989 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:32:56.989 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:32:56.989 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:32:56.990 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:32:56.990 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:32:57.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:32:57.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:32:57.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:32:57.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:32:57.198 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:32:57.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:32:57.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:32:57.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:32:57.617 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:32:57.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836490 2023-09-30T01:32:57.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836490 2023-09-30T01:32:57.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836490' 2023-09-30T01:32:57.765 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:32:57.766 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836490 2023-09-30T01:32:57.766 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:32:57.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:32:57.769 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836490 2023-09-30T01:32:57.769 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:32:57.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836490 2023-09-30T01:32:57.771 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836490 2023-09-30T01:32:57.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2023-09-30T01:32:57.774 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:32:58.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836488 -lt 21474836490 2023-09-30T01:32:58.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:32:59.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:32:59.176 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:32:59.611 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836491 -lt 21474836490 2023-09-30T01:32:59.612 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:32:59.612 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:32:59.612 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:33:00.149 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:33:00.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:33:00.150 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:33:00.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:33:00.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:33:00.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:33:00.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:33:00.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:33:00.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:33:00.537 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:33:00.538 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:33:00.538 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:33:01.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:33:01.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:33:01.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:33:01.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1316: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2023-09-30T01:33:01.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2023-09-30T01:33:01.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2023-09-30T01:33:01.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2023-09-30T01:33:01.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: test_objectstore_tool: diff - /etc/group 2023-09-30T01:33:01.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2023-09-30T01:33:01.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2023-09-30T01:33:01.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2023-09-30T01:33:01.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2023-09-30T01:33:01.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2023-09-30T01:33:01.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2023-09-30T01:33:01.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2023-09-30T01:33:01.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2023-09-30T01:33:01.200 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:33:01.200 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:33:01.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:33:01.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:33:01.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:33:01.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:33:01.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2023-09-30T01:33:01.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2023-09-30T01:33:01.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2023-09-30T01:33:01.513 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2023-09-30T01:33:01.514 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2023-09-30T01:33:01.514 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2023-09-30T01:33:01.514 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2023-09-30T01:33:02.847 INFO:tasks.workunit.client.0.smithi186.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2023-09-30T01:33:02.849 INFO:tasks.workunit.client.0.smithi186.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2023-09-30T01:33:02.850 INFO:tasks.workunit.client.0.smithi186.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2023-09-30T01:33:02.853 INFO:tasks.workunit.client.0.smithi186.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2023-09-30T01:33:03.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2023-09-30T01:33:03.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2023-09-30T01:33:03.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-09-30T01:33:03.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-09-30T01:33:03.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-09-30T01:33:03.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:33:03.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:33:03.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:33:03.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:33:03.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:33:03.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:33:03.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:33:03.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-09-30T01:33:03.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-09-30T01:33:03.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:33:03.165 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-09-30T01:33:03.165 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:33:03.165 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:33:03.167 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:33:03.167 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:33:03.168 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:33:03.168 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:33:03.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:33:03.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:33:03.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:33:03.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:33:03.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:33:03.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:33:03.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:33:03.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:33:03.169 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-09-30T01:33:03.169 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-09-30T01:33:03.169 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:33:03.170 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-09-30T01:33:03.170 INFO:tasks.workunit.client.0.smithi186.stderr:start osd.0 2023-09-30T01:33:03.170 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:33:03.170 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2023-09-30T01:33:03.170 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-09-30T01:33:03.171 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-09-30T01:33:03.171 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-09-30T01:33:03.172 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-09-30T01:33:03.200 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:03.196+0000 7f01558803c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:33:03.203 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:03.196+0000 7f01558803c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:33:03.205 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:03.200+0000 7f01558803c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:33:03.514 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:03.508+0000 7f01558803c0 -1 Falling back to public interface 2023-09-30T01:33:03.576 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-09-30T01:33:03.576 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:33:03.576 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:33:03.576 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:33:03.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:33:03.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:03.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:33:03.577 INFO:tasks.workunit.client.0.smithi186.stderr:0 2023-09-30T01:33:03.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:03.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:03.979 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:04.981 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:04.981 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:04.981 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:33:04.981 INFO:tasks.workunit.client.0.smithi186.stderr:1 2023-09-30T01:33:04.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:04.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:05.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:06.162 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:06.160+0000 7f01558803c0 -1 osd.0 13 log_to_monitors true 2023-09-30T01:33:06.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:06.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:06.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:33:06.385 INFO:tasks.workunit.client.0.smithi186.stderr:2 2023-09-30T01:33:06.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:06.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:06.828 INFO:tasks.workunit.client.0.smithi186.stderr:osd.0 up in weight 1 up_from 16 up_thru 7 down_at 14 last_clean_interval [5,13) [v2:127.0.0.1:6802/130869,v1:127.0.0.1:6803/130869] [v2:127.0.0.1:6804/130869,v1:127.0.0.1:6805/130869] exists,up f4004270-c8ac-46b0-99d8-6fb1012a2f42 2023-09-30T01:33:06.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:33:06.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:33:06.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:33:06.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2023-09-30T01:33:06.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:33:06.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:33:06.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:33:06.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:33:06.830 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:33:06.830 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:33:06.830 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:33:06.830 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:33:06.830 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:33:06.830 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:33:07.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:33:07.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:33:07.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:33:07.029 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:33:07.029 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:33:07.434 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:33:07.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:33:07.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:33:07.435 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:33:07.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=68719476739 2023-09-30T01:33:07.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 68719476739 2023-09-30T01:33:07.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-68719476739' 2023-09-30T01:33:07.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:33:07.623 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-68719476739 2023-09-30T01:33:07.623 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:33:07.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:33:07.625 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-68719476739 2023-09-30T01:33:07.625 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:33:07.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=68719476739 2023-09-30T01:33:07.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 68719476739' 2023-09-30T01:33:07.627 INFO:tasks.workunit.client.0.smithi186.stderr:waiting osd.0 seq 68719476739 2023-09-30T01:33:07.628 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:33:08.050 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 1 -lt 68719476739 2023-09-30T01:33:08.050 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:33:09.052 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:33:09.052 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:33:09.466 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 1 -lt 68719476739 2023-09-30T01:33:09.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:33:10.468 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 299 -eq 0 ']' 2023-09-30T01:33:10.469 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:33:10.936 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 68719476739 -lt 68719476739 2023-09-30T01:33:10.936 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:33:10.937 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:33:10.937 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:33:11.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:33:11.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:33:11.445 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:33:11.445 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:33:11.446 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:33:11.446 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:33:11.446 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:33:11.446 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:33:11.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:33:11.845 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:33:11.846 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:33:11.846 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:33:12.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:33:12.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:33:12.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:33:12.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1319: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2023-09-30T01:33:12.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2023-09-30T01:33:12.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2023-09-30T01:33:12.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2023-09-30T01:33:12.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2023-09-30T01:33:12.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2023-09-30T01:33:12.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2023-09-30T01:33:12.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2023-09-30T01:33:12.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2023-09-30T01:33:12.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2023-09-30T01:33:12.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2023-09-30T01:33:12.327 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:33:12.327 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:33:12.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:33:12.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:33:12.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:33:12.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:33:12.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2023-09-30T01:33:12.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2023-09-30T01:33:12.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2023-09-30T01:33:12.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2023-09-30T01:33:12.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2023-09-30T01:33:12.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2023-09-30T01:33:12.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2023-09-30T01:33:13.629 INFO:tasks.workunit.client.0.smithi186.stderr:Error getting attr on : 1.3_head,#1:c0000000::::head#, (61) No data available 2023-09-30T01:33:13.630 INFO:tasks.workunit.client.0.smithi186.stderr:Error getting attr on : 1.2_head,#1:40000000::::head#, (61) No data available 2023-09-30T01:33:13.631 INFO:tasks.workunit.client.0.smithi186.stderr:Error getting attr on : 1.1_head,#1:80000000::::head#, (61) No data available 2023-09-30T01:33:13.632 INFO:tasks.workunit.client.0.smithi186.stderr:Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available 2023-09-30T01:33:13.632 INFO:tasks.workunit.client.0.smithi186.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2023-09-30T01:33:13.931 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2023-09-30T01:33:13.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: return 1 2023-09-30T01:33:13.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: return 1 2023-09-30T01:33:13.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1320: test_objectstore_tool: teardown td/ceph-helpers 2023-09-30T01:33:13.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:33:13.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:33:13.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:33:13.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:33:13.933 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:33:13.933 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:33:13.933 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:33:13.933 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:33:14.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:33:14.047 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:33:14.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:33:14.049 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:33:14.051 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:33:14.051 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:33:14.052 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:33:14.054 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:33:14.054 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:33:14.054 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:33:14.055 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:33:14.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:33:14.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:33:14.059 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:33:14.069 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:33:14.070 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:33:14.070 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:33:14.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:33:14.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:33:14.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:33:14.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:33:14.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_pg_scrub td/ceph-helpers 2023-09-30T01:33:14.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1863: test_pg_scrub: local dir=td/ceph-helpers 2023-09-30T01:33:14.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1865: test_pg_scrub: setup td/ceph-helpers 2023-09-30T01:33:14.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:33:14.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:33:14.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:33:14.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:33:14.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:33:14.074 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:33:14.074 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:33:14.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:33:14.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:33:14.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:33:14.077 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:33:14.078 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:33:14.080 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:33:14.080 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:33:14.082 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:33:14.082 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:33:14.083 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:33:14.084 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:33:14.084 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:33:14.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:33:14.085 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:33:14.086 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:33:14.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:33:14.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:33:14.090 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:33:14.091 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:33:14.091 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:33:14.091 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:33:14.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:33:14.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:33:14.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:33:14.095 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:33:14.095 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:33:14.095 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:33:14.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:33:14.097 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:33:14.099 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:33:14.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:33:14.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:33:14.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1866: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:33:14.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:33:14.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:33:14.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:33:14.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:33:14.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:33:14.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:33:14.531 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:33:14.531 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:33:14.532 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:33:14.532 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:33:14.532 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:33:14.532 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:33:14.533 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:33:14.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:33:14.834 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:33:14.836 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:33:14.836 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:33:14.836 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:33:14.836 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:33:14.837 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:33:14.838 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:33:14.838 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:33:14.838 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:33:14.838 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:33:14.839 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:33:14.839 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:33:14.839 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:33:14.839 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:33:14.840 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:33:14.935 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:33:14.935 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:33:14.935 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:33:14.936 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:33:14.937 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:33:14.937 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:33:14.937 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:33:14.937 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:33:14.937 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:33:14.937 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:33:14.937 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:33:14.937 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:33:14.938 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:33:14.938 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:33:15.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1867: test_pg_scrub: run_mgr td/ceph-helpers x 2023-09-30T01:33:15.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:33:15.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:33:15.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:33:15.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:33:15.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:33:15.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:33:15.927 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:33:15.928 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:33:15.928 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:33:15.929 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:33:15.929 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:33:15.929 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:33:15.929 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:33:15.930 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:33:15.931 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:33:15.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1868: test_pg_scrub: run_osd td/ceph-helpers 0 2023-09-30T01:33:15.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:33:15.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:33:15.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:33:15.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:33:15.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:33:15.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:33:15.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:33:15.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:33:15.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:33:15.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:33:15.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:33:15.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:33:15.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:33:15.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:33:15.966 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:33:15.966 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:33:15.966 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:33:15.966 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:33:15.966 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:33:15.966 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:33:15.966 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:33:15.966 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:33:15.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:33:15.968 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:33:15.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0800284a-013f-4bd7-9e9e-cd2f1538c9aa 2023-09-30T01:33:15.968 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 0800284a-013f-4bd7-9e9e-cd2f1538c9aa 2023-09-30T01:33:15.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0800284a-013f-4bd7-9e9e-cd2f1538c9aa' 2023-09-30T01:33:15.969 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:33:15.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbehdlXUjoOhAAdOXj725+XDAecddpFqU8Mw== 2023-09-30T01:33:15.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbehdlXUjoOhAAdOXj725+XDAecddpFqU8Mw=="}' 2023-09-30T01:33:15.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0800284a-013f-4bd7-9e9e-cd2f1538c9aa -i td/ceph-helpers/0/new.json 2023-09-30T01:33:16.386 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:33:16.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:33:16.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbehdlXUjoOhAAdOXj725+XDAecddpFqU8Mw== --osd-uuid 0800284a-013f-4bd7-9e9e-cd2f1538c9aa 2023-09-30T01:33:16.432 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:16.431+0000 7f540fb2b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:33:16.436 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:16.435+0000 7f540fb2b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:33:16.437 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:16.435+0000 7f540fb2b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:33:16.438 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:16.435+0000 7f540fb2b3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:33:16.439 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:16.435+0000 7f540fb2b3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:33:16.439 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:16.435+0000 7f540fb2b3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:33:16.526 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:16.523+0000 7f540fb2b3c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:33:23.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:33:23.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:33:23.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:33:23.012 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:33:23.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:33:23.555 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:33:23.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:33:23.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:33:23.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:33:23.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:33:23.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:33:23.587 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:23.583+0000 7f171ac5b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:33:23.589 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:23.587+0000 7f171ac5b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:33:23.590 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:23.587+0000 7f171ac5b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:33:23.879 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:23.875+0000 7f171ac5b3c0 -1 Falling back to public interface 2023-09-30T01:33:23.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:33:23.956 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:33:23.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:33:23.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:33:23.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:33:23.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:23.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:33:23.958 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:33:23.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:23.959 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:24.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:25.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:25.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:25.385 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:33:25.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:33:25.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:25.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:25.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:26.500 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:33:26.499+0000 7f171ac5b3c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:33:26.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:26.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:26.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:33:26.805 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:33:26.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:26.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:27.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:28.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:28.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:28.339 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:33:28.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:33:28.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:28.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:28.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:29.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:29.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:29.789 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:33:29.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:33:29.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:29.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:30.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:31.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:31.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:31.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:33:31.391 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:33:31.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:31.392 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:31.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:32.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:32.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:32.825 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:33:32.826 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:33:32.826 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:32.826 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:33.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:34.230 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:34.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:34.231 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:33:34.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:33:34.232 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:34.232 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:34.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:35.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:35.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:35.622 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:33:35.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:33:35.622 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:35.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:36.037 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:37.039 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:37.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:37.040 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:33:37.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:33:37.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:37.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:37.638 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:38.639 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:38.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:38.640 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:33:38.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:33:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:38.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:39.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:40.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:40.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:40.095 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:33:40.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:33:40.096 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:40.097 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:40.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:41.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:41.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:41.624 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:33:41.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:33:41.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:41.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:42.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:43.606 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:43.606 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:43.606 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:33:43.607 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:33:43.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:43.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:44.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:45.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:45.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:45.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:33:45.189 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:33:45.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:45.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:45.941 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:46.943 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:46.943 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:46.943 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:33:46.944 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:33:46.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:46.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:47.991 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:33:48.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:33:48.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:33:48.993 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:33:48.994 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:33:48.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:33:48.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:33:49.401 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/133252,v1:127.0.0.1:6803/133252] [v2:127.0.0.1:6804/133252,v1:127.0.0.1:6805/133252] exists,up 0800284a-013f-4bd7-9e9e-cd2f1538c9aa 2023-09-30T01:33:49.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:33:49.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:33:49.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:33:49.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1869: test_pg_scrub: create_rbd_pool 2023-09-30T01:33:49.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:33:49.848 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:33:49.856 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:33:49.856 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:33:50.564 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:33:50.575 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:33:51.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:33:52.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1870: test_pg_scrub: wait_for_clean 2023-09-30T01:33:52.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:33:52.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:33:52.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:33:52.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:33:52.646 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:33:52.647 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:33:52.647 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:33:52.647 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:33:52.648 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:33:52.648 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:33:52.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:33:52.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:33:52.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:33:52.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:33:52.858 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:33:53.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:33:53.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:33:53.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:33:53.326 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:33:53.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836488 2023-09-30T01:33:53.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836488 2023-09-30T01:33:53.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836488' 2023-09-30T01:33:53.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:33:53.485 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836488 2023-09-30T01:33:53.485 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:33:53.487 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:33:53.488 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836488 2023-09-30T01:33:53.488 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:33:53.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836488 2023-09-30T01:33:53.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2023-09-30T01:33:53.490 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836488 2023-09-30T01:33:53.490 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:33:53.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836486 -lt 21474836488 2023-09-30T01:33:53.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:33:54.896 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:33:54.896 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:33:55.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836489 -lt 21474836488 2023-09-30T01:33:55.315 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:33:55.316 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:33:55.316 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:33:55.913 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:33:55.913 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:33:55.914 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:33:55.914 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:33:55.914 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:33:55.914 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:33:55.914 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:33:55.914 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:33:56.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:33:56.325 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:33:56.325 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:33:56.325 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:33:56.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:33:56.831 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:33:56.831 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:33:56.831 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1871: test_pg_scrub: pg_scrub 1.0 2023-09-30T01:33:56.831 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: pg_scrub: local pgid=1.0 2023-09-30T01:33:56.832 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: pg_scrub: get_last_scrub_stamp 1.0 2023-09-30T01:33:56.832 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:33:56.832 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:33:56.832 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:33:56.832 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:33:57.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: pg_scrub: local last_scrub=2023-09-30T01:33:50.282929+0000 2023-09-30T01:33:57.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1851: pg_scrub: ceph pg scrub 1.0 2023-09-30T01:33:57.617 INFO:tasks.workunit.client.0.smithi186.stderr:instructing pg 1.0 on osd.0 to scrub 2023-09-30T01:33:57.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1852: pg_scrub: wait_for_scrub 1.0 2023-09-30T01:33:50.282929+0000 2023-09-30T01:33:57.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: wait_for_scrub: local pgid=1.0 2023-09-30T01:33:57.625 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: wait_for_scrub: local last_scrub=2023-09-30T01:33:50.282929+0000 2023-09-30T01:33:57.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_for_scrub: local sname=last_scrub_stamp 2023-09-30T01:33:57.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i=0 )) 2023-09-30T01:33:57.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2023-09-30T01:33:57.627 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2023-09-30T01:33:57.627 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:33:57.627 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:33:57.627 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:33:57.627 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:33:58.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2023-09-30T01:33:50.282929+0000 '>' 2023-09-30T01:33:50.282929+0000 2023-09-30T01:33:58.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2023-09-30T01:33:59.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2023-09-30T01:33:59.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2023-09-30T01:33:59.003 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2023-09-30T01:33:59.003 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:33:59.003 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:33:59.004 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:33:59.004 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:33:59.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2023-09-30T01:33:58.716563+0000 '>' 2023-09-30T01:33:50.282929+0000 2023-09-30T01:33:59.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_for_scrub: return 0 2023-09-30T01:33:59.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2023-09-30T01:33:59.448 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:33:59.448 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:33:59.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:33:59.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:33:59.448 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:33:59.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:33:59.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_pg_scrub: TIMEOUT=1 2023-09-30T01:33:59.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: test_pg_scrub: pg_scrub 1.0 2023-09-30T01:33:59.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: pg_scrub: local pgid=1.0 2023-09-30T01:33:59.560 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: pg_scrub: get_last_scrub_stamp 1.0 2023-09-30T01:33:59.561 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:33:59.561 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:33:59.561 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:33:59.561 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:33:59.943 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: pg_scrub: local last_scrub=2023-09-30T01:33:58.716563+0000 2023-09-30T01:33:59.943 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1851: pg_scrub: ceph pg scrub 1.0 2023-09-30T01:34:00.349 INFO:tasks.workunit.client.0.smithi186.stderr:instructing pg 1.0 on osd.0 to scrub 2023-09-30T01:34:00.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1852: pg_scrub: wait_for_scrub 1.0 2023-09-30T01:33:58.716563+0000 2023-09-30T01:34:00.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: wait_for_scrub: local pgid=1.0 2023-09-30T01:34:00.363 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: wait_for_scrub: local last_scrub=2023-09-30T01:33:58.716563+0000 2023-09-30T01:34:00.364 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_for_scrub: local sname=last_scrub_stamp 2023-09-30T01:34:00.364 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i=0 )) 2023-09-30T01:34:00.364 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 1 )) 2023-09-30T01:34:00.365 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2023-09-30T01:34:00.365 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:34:00.365 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:34:00.365 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:34:00.365 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:34:00.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2023-09-30T01:33:58.716563+0000 '>' 2023-09-30T01:33:58.716563+0000 2023-09-30T01:34:00.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2023-09-30T01:34:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2023-09-30T01:34:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 1 )) 2023-09-30T01:34:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: wait_for_scrub: return 1 2023-09-30T01:34:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1874: test_pg_scrub: teardown td/ceph-helpers 2023-09-30T01:34:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:34:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:34:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:34:01.727 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:34:01.727 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:34:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:34:01.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:34:01.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:34:01.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:34:01.843 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:34:01.845 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:34:01.846 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:34:01.848 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:34:01.848 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:34:01.849 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:34:01.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:34:01.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:34:01.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:34:01.852 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:34:01.853 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:34:01.855 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:34:01.855 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:34:01.866 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:34:01.866 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:01.866 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:01.866 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:34:01.868 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:34:01.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:34:01.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:34:01.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_repair td/ceph-helpers 2023-09-30T01:34:01.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1823: test_repair: local dir=td/ceph-helpers 2023-09-30T01:34:01.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1825: test_repair: setup td/ceph-helpers 2023-09-30T01:34:01.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:34:01.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:34:01.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:34:01.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:34:01.869 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:34:01.870 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:34:01.870 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:34:01.870 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:34:01.870 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:34:01.870 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:34:01.873 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:34:01.874 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:34:01.875 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:34:01.876 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:34:01.877 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:34:01.877 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:34:01.878 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:34:01.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:34:01.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:34:01.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:34:01.880 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:34:01.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:34:01.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:34:01.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:34:01.885 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:34:01.885 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:01.885 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:01.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:34:01.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:34:01.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:34:01.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:34:01.889 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:34:01.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:01.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:01.890 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:34:01.892 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:34:01.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:34:01.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:34:01.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:34:01.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1826: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:34:01.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:34:01.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:34:01.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:34:01.894 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:34:01.895 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:34:01.895 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:34:02.278 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:34:02.279 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:34:02.279 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:34:02.279 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:34:02.280 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:02.280 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:02.280 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:34:02.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:34:02.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:34:02.620 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:34:02.620 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:34:02.620 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:34:02.621 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:34:02.621 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:34:02.621 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:34:02.622 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:34:02.622 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:34:02.622 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:34:02.622 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:02.623 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:02.623 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:34:02.623 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:34:02.624 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:34:02.716 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:34:02.716 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:34:02.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:34:02.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:34:02.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:34:02.718 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:34:02.718 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:34:02.718 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:34:02.718 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:34:02.719 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:02.719 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:02.719 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:34:02.719 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:34:02.719 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:34:02.799 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1827: test_repair: run_mgr td/ceph-helpers x 2023-09-30T01:34:02.799 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:34:02.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:34:02.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:34:02.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:34:02.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:34:02.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:34:03.700 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:34:03.700 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:34:03.700 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:34:03.700 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:34:03.700 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:03.701 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:03.701 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:34:03.702 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:34:03.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:34:03.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1828: test_repair: run_osd td/ceph-helpers 0 2023-09-30T01:34:03.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:34:03.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:34:03.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:34:03.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:34:03.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:34:03.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:34:03.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:34:03.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:34:03.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:34:03.740 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:34:03.741 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:34:03.741 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:34:03.741 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:34:03.741 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:34:03.741 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:34:03.741 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:34:03.741 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:34:03.742 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:34:03.742 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:03.743 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:03.743 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:34:03.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:34:03.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:34:03.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:34:03.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:34:03.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:34:03.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:34:03.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:34:03.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:34:03.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:34:03.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:34:03.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:34:03.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:34:03.744 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:34:03.744 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:34:03.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d4afbdd8-0105-4620-a80a-3609448f32bf 2023-09-30T01:34:03.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d4afbdd8-0105-4620-a80a-3609448f32bf' 2023-09-30T01:34:03.745 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 d4afbdd8-0105-4620-a80a-3609448f32bf 2023-09-30T01:34:03.746 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:34:03.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALexdlPgudLRAAuG3g2+qaqiuPWoOjBGgJcQ== 2023-09-30T01:34:03.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALexdlPgudLRAAuG3g2+qaqiuPWoOjBGgJcQ=="}' 2023-09-30T01:34:03.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d4afbdd8-0105-4620-a80a-3609448f32bf -i td/ceph-helpers/0/new.json 2023-09-30T01:34:04.128 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:34:04.141 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:34:04.142 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALexdlPgudLRAAuG3g2+qaqiuPWoOjBGgJcQ== --osd-uuid d4afbdd8-0105-4620-a80a-3609448f32bf 2023-09-30T01:34:04.175 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:04.171+0000 7f1ef53c13c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:34:04.177 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:04.175+0000 7f1ef53c13c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:34:04.179 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:04.175+0000 7f1ef53c13c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:34:04.180 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:04.179+0000 7f1ef53c13c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:34:04.180 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:04.179+0000 7f1ef53c13c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:34:04.180 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:04.179+0000 7f1ef53c13c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:34:04.259 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:04.255+0000 7f1ef53c13c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:34:10.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:34:10.822 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:34:10.824 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:34:10.824 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:34:10.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:34:11.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:34:11.352 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:34:11.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:34:11.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:34:11.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:34:11.353 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:34:11.384 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:11.383+0000 7f59065093c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:34:11.386 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:11.383+0000 7f59065093c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:34:11.388 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:11.387+0000 7f59065093c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:34:11.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:34:11.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:34:11.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:34:11.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:34:11.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:34:11.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:11.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:34:11.803 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:34:11.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:11.803 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:11.950 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:11.947+0000 7f59065093c0 -1 Falling back to public interface 2023-09-30T01:34:12.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:13.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:13.252 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:13.252 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:34:13.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:34:13.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:13.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:13.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:14.475 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:14.470+0000 7f59065093c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:34:14.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:14.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:14.657 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:34:14.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:34:14.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:14.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:15.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:16.178 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:16.179 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:16.179 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:34:16.179 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:34:16.179 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:16.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:16.583 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:17.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:17.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:17.586 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:34:17.586 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:34:17.586 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:17.587 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:18.132 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:19.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:19.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:19.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:34:19.134 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:34:19.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:19.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:19.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:20.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:20.557 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:20.558 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:34:20.558 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:34:20.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:20.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:20.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:21.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:21.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:21.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:34:21.968 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:34:21.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:21.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:22.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:23.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:23.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:23.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:34:23.402 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:34:23.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:23.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:23.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:24.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:24.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:24.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:34:24.812 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:34:24.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:24.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:25.301 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:26.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:26.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:26.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:34:26.304 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:34:26.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:26.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:27.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:28.121 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:28.122 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:28.122 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:34:28.122 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:34:28.122 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:28.122 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:28.621 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:29.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:29.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:29.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:34:29.623 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:34:29.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:29.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:30.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:31.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:31.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:31.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:34:31.619 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:34:31.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:31.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:32.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:33.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:33.118 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:33.118 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:34:33.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:34:33.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:33.119 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:33.927 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:34.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:34.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:34.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:34:34.929 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:34:34.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:34.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:35.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:34:36.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:34:36.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:34:36.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:34:36.789 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:34:36.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:34:36.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:34:37.228 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/135837,v1:127.0.0.1:6803/135837] [v2:127.0.0.1:6804/135837,v1:127.0.0.1:6805/135837] exists,up d4afbdd8-0105-4620-a80a-3609448f32bf 2023-09-30T01:34:37.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:34:37.228 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:34:37.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:34:37.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1829: test_repair: create_rbd_pool 2023-09-30T01:34:37.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:34:37.730 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:34:37.741 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:34:37.742 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:34:38.410 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:34:38.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:34:39.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:34:40.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: test_repair: wait_for_clean 2023-09-30T01:34:40.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:34:40.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:34:40.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:34:40.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:34:40.564 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:34:40.564 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:34:40.564 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:34:40.565 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:34:40.565 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:34:40.565 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:34:40.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:34:40.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:34:40.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:34:40.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:34:40.767 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:34:41.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:34:41.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:34:41.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:34:41.197 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:34:41.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836487 2023-09-30T01:34:41.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836487 2023-09-30T01:34:41.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836487' 2023-09-30T01:34:41.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:34:41.375 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836487 2023-09-30T01:34:41.375 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:34:41.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:34:41.378 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836487 2023-09-30T01:34:41.379 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:34:41.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836487 2023-09-30T01:34:41.381 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2023-09-30T01:34:41.381 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836487 2023-09-30T01:34:41.381 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:34:41.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836485 -lt 21474836487 2023-09-30T01:34:41.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:34:42.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:34:42.788 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:34:43.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836488 -lt 21474836487 2023-09-30T01:34:43.190 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:34:43.191 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:34:43.191 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:34:43.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:34:43.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:34:43.757 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:34:43.757 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:34:43.757 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:34:43.757 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:34:43.758 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:34:43.758 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:34:44.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:34:44.247 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:34:44.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:34:44.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:34:44.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:34:44.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:34:44.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:34:44.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1831: test_repair: repair 1.0 2023-09-30T01:34:44.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1816: repair: local pgid=1.0 2023-09-30T01:34:44.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1817: repair: get_last_scrub_stamp 1.0 2023-09-30T01:34:44.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:34:44.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:34:44.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:34:44.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:34:45.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1817: repair: local last_scrub=2023-09-30T01:34:38.089390+0000 2023-09-30T01:34:45.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: repair: ceph pg repair 1.0 2023-09-30T01:34:45.498 INFO:tasks.workunit.client.0.smithi186.stderr:instructing pg 1.0 on osd.0 to repair 2023-09-30T01:34:45.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1819: repair: wait_for_scrub 1.0 2023-09-30T01:34:38.089390+0000 2023-09-30T01:34:45.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: wait_for_scrub: local pgid=1.0 2023-09-30T01:34:45.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: wait_for_scrub: local last_scrub=2023-09-30T01:34:38.089390+0000 2023-09-30T01:34:45.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_for_scrub: local sname=last_scrub_stamp 2023-09-30T01:34:45.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i=0 )) 2023-09-30T01:34:45.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2023-09-30T01:34:45.507 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2023-09-30T01:34:45.507 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:34:45.507 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:34:45.507 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:34:45.507 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:34:45.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2023-09-30T01:34:38.089390+0000 '>' 2023-09-30T01:34:38.089390+0000 2023-09-30T01:34:45.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2023-09-30T01:34:46.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2023-09-30T01:34:46.889 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2023-09-30T01:34:46.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2023-09-30T01:34:46.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:34:46.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:34:46.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:34:46.891 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:34:47.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2023-09-30T01:34:38.089390+0000 '>' 2023-09-30T01:34:38.089390+0000 2023-09-30T01:34:47.300 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2023-09-30T01:34:48.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2023-09-30T01:34:48.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2023-09-30T01:34:48.303 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2023-09-30T01:34:48.303 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:34:48.303 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:34:48.303 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:34:48.303 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:34:48.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2023-09-30T01:34:38.089390+0000 '>' 2023-09-30T01:34:38.089390+0000 2023-09-30T01:34:48.700 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2023-09-30T01:34:49.701 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2023-09-30T01:34:49.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 300 )) 2023-09-30T01:34:49.702 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2023-09-30T01:34:49.702 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:34:49.702 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:34:49.703 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:34:49.703 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:34:50.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2023-09-30T01:34:46.588655+0000 '>' 2023-09-30T01:34:38.089390+0000 2023-09-30T01:34:50.127 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: wait_for_scrub: return 0 2023-09-30T01:34:50.127 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: test_repair: kill_daemons td/ceph-helpers KILL osd 2023-09-30T01:34:50.128 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:34:50.128 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:34:50.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:34:50.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:34:50.128 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:34:50.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:34:50.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: test_repair: TIMEOUT=1 2023-09-30T01:34:50.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: test_repair: repair 1.0 2023-09-30T01:34:50.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1816: repair: local pgid=1.0 2023-09-30T01:34:50.238 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1817: repair: get_last_scrub_stamp 1.0 2023-09-30T01:34:50.238 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:34:50.238 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:34:50.238 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:34:50.238 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:34:50.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1817: repair: local last_scrub=2023-09-30T01:34:46.588655+0000 2023-09-30T01:34:50.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: repair: ceph pg repair 1.0 2023-09-30T01:34:51.004 INFO:tasks.workunit.client.0.smithi186.stderr:instructing pg 1.0 on osd.0 to repair 2023-09-30T01:34:51.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1819: repair: wait_for_scrub 1.0 2023-09-30T01:34:46.588655+0000 2023-09-30T01:34:51.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: wait_for_scrub: local pgid=1.0 2023-09-30T01:34:51.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: wait_for_scrub: local last_scrub=2023-09-30T01:34:46.588655+0000 2023-09-30T01:34:51.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: wait_for_scrub: local sname=last_scrub_stamp 2023-09-30T01:34:51.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i=0 )) 2023-09-30T01:34:51.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 1 )) 2023-09-30T01:34:51.013 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2023-09-30T01:34:51.013 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2023-09-30T01:34:51.013 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2023-09-30T01:34:51.014 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2023-09-30T01:34:51.014 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2023-09-30T01:34:51.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: wait_for_scrub: test 2023-09-30T01:34:46.588655+0000 '>' 2023-09-30T01:34:46.588655+0000 2023-09-30T01:34:51.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: wait_for_scrub: sleep 1 2023-09-30T01:34:52.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i++ )) 2023-09-30T01:34:52.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: wait_for_scrub: (( i < 1 )) 2023-09-30T01:34:52.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: wait_for_scrub: return 1 2023-09-30T01:34:52.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: test_repair: teardown td/ceph-helpers 2023-09-30T01:34:52.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:34:52.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:34:52.405 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:34:52.405 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:34:52.405 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:34:52.406 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:34:52.406 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:34:52.406 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:34:52.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:34:52.520 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:34:52.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:34:52.523 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:34:52.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:34:52.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:34:52.526 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:34:52.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:34:52.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:34:52.528 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:34:52.529 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:34:52.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:34:52.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:34:52.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:34:52.541 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:34:52.541 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:52.541 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:52.542 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:34:52.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:34:52.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:34:52.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:34:52.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_run_in_background td/ceph-helpers 2023-09-30T01:34:52.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: test_run_in_background: local pids 2023-09-30T01:34:52.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: test_run_in_background: run_in_background pids sleep 1 2023-09-30T01:34:52.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 137364"' 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 137364' 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: test_run_in_background: run_in_background pids sleep 1 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/137366: /' 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sleep 1 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 137367"' 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 137367' 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sleep 1 2023-09-30T01:34:52.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/137370: /' 2023-09-30T01:34:52.549 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: test_run_in_background: echo 137364 137367 2023-09-30T01:34:52.549 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: test_run_in_background: wc -w 2023-09-30T01:34:52.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: test_run_in_background: test 2 = 2 2023-09-30T01:34:52.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: test_run_in_background: wait 137364 137367 2023-09-30T01:34:53.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 0 2023-09-30T01:34:53.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 0 2023-09-30T01:34:53.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:34:53.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_run_mon td/ceph-helpers 2023-09-30T01:34:53.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:495: test_run_mon: local dir=td/ceph-helpers 2023-09-30T01:34:53.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:497: test_run_mon: setup td/ceph-helpers 2023-09-30T01:34:53.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:34:53.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:34:53.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:34:53.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:34:53.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:34:53.550 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:34:53.550 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:34:53.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:34:53.550 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:34:53.551 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:34:53.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:34:53.555 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:34:53.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:34:53.557 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:34:53.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:34:53.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:34:53.560 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:34:53.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:34:53.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:34:53.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:34:53.563 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:34:53.564 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:34:53.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:34:53.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:34:53.568 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:34:53.568 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:53.568 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:53.568 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:34:53.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:34:53.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:34:53.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:34:53.573 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:34:53.573 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:53.573 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:53.573 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:34:53.576 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:34:53.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:34:53.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:34:53.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:34:53.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: run_mon td/ceph-helpers a 2023-09-30T01:34:53.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:34:53.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:34:53.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:34:53.577 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:34:53.578 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:34:53.578 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2023-09-30T01:34:53.967 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:34:53.967 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:34:53.967 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:34:53.968 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:34:53.968 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:53.968 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:53.968 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:34:53.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:34:54.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:34:54.309 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:34:54.309 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:34:54.309 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:34:54.309 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:34:54.310 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:34:54.310 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:34:54.311 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:34:54.311 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:34:54.311 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:34:54.311 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:54.311 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:54.312 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:34:54.312 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:34:54.313 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:34:54.403 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:34:54.403 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:34:54.403 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:34:54.403 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:34:54.404 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:34:54.404 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:34:54.404 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:34:54.404 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:34:54.405 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:34:54.405 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:54.405 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:54.405 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:34:54.406 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:34:54.406 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:34:54.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: ceph mon dump 2023-09-30T01:34:54.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: grep mon.a 2023-09-30T01:34:55.328 INFO:tasks.workunit.client.0.smithi186.stderr:dumped monmap epoch 1 2023-09-30T01:34:55.336 INFO:tasks.workunit.client.0.smithi186.stdout:0: v2:127.0.0.1:7109/0 mon.a 2023-09-30T01:34:55.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: kill_daemons td/ceph-helpers 2023-09-30T01:34:55.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:34:55.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:34:55.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:34:55.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:34:55.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:34:56.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:34:56.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:503: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=3 2023-09-30T01:34:56.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:34:56.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:34:56.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:34:56.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:34:56.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:34:56.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=3 2023-09-30T01:34:56.680 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:56.678+0000 7f04ae8e3a00 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2023-09-30T01:34:56.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:34:56.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:34:56.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:34:56.683 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:34:56.683 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:56.683 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:56.684 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:34:56.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2023-09-30T01:34:56.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:34:56.967 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:34:56.967 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:34:56.967 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:34:56.967 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:34:56.967 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:34:56.969 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:34:56.969 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:34:56.969 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:34:56.969 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:34:56.970 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:56.970 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:56.970 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:34:56.970 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:34:56.971 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:34:57.061 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:34:57.061 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:34:57.061 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:34:57.061 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:34:57.062 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:34:57.062 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:34:57.062 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:34:57.062 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:34:57.063 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:34:57.063 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:57.063 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:57.063 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:34:57.064 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:34:57.064 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:34:57.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: run_osd td/ceph-helpers 0 2023-09-30T01:34:57.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:34:57.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:34:57.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:34:57.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:34:57.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:34:57.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:34:57.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:34:57.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:34:57.151 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:34:57.151 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:34:57.151 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:34:57.151 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:34:57.151 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:34:57.151 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:34:57.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:34:57.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:34:57.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:34:57.153 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:34:57.153 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:34:57.153 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:34:57.153 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:34:57.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:34:57.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:34:57.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:34:57.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:34:57.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:34:57.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:34:57.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:34:57.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:34:57.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:34:57.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:34:57.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:34:57.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:34:57.154 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:34:57.155 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:34:57.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ea1bd145-b197-4f6e-83b1-e76adbed579e 2023-09-30T01:34:57.155 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ea1bd145-b197-4f6e-83b1-e76adbed579e' 2023-09-30T01:34:57.155 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 ea1bd145-b197-4f6e-83b1-e76adbed579e 2023-09-30T01:34:57.156 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:34:57.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBexdl5chzChAAXJ7rHsN8NAlaJJeMkLMi5w== 2023-09-30T01:34:57.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBexdl5chzChAAXJ7rHsN8NAlaJJeMkLMi5w=="}' 2023-09-30T01:34:57.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ea1bd145-b197-4f6e-83b1-e76adbed579e -i td/ceph-helpers/0/new.json 2023-09-30T01:34:57.545 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:34:57.560 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:34:57.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBexdl5chzChAAXJ7rHsN8NAlaJJeMkLMi5w== --osd-uuid ea1bd145-b197-4f6e-83b1-e76adbed579e 2023-09-30T01:34:57.596 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:57.594+0000 7fdd19c233c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:34:57.599 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:57.594+0000 7fdd19c233c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:34:57.602 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:57.598+0000 7fdd19c233c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:34:57.603 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:57.598+0000 7fdd19c233c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:34:57.603 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:57.598+0000 7fdd19c233c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:34:57.603 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:57.598+0000 7fdd19c233c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:34:57.697 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:34:57.694+0000 7fdd19c233c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:35:03.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:35:03.823 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:35:03.825 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:35:03.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:35:03.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:35:04.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:35:04.063 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:35:04.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:35:04.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:35:04.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:35:04.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:35:04.088 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:04.086+0000 7f7bd00f33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:35:04.103 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:04.098+0000 7f7bd00f33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:35:04.105 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:04.102+0000 7f7bd00f33c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:35:04.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:35:04.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:35:04.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:35:04.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:35:04.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:35:04.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:04.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:35:04.297 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:35:04.298 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:04.298 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:04.572 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:04.650 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:04.646+0000 7f7bd00f33c0 -1 Falling back to public interface 2023-09-30T01:35:05.573 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:05.574 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:05.574 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:35:05.574 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:35:05.574 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:05.575 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:05.799 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:06.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:06.801 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:06.801 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:35:06.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:35:06.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:06.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:07.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:07.329 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:07.326+0000 7f7bd00f33c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:35:08.044 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:08.044 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:08.045 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:35:08.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:35:08.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:08.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:08.467 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:09.469 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:09.484 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:09.485 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:35:09.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:35:09.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:09.485 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:09.700 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:10.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:10.702 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:10.703 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:35:10.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:35:10.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:10.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:10.932 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:11.934 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:11.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:11.935 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:35:11.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:35:11.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:11.936 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:12.194 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:13.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:13.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:13.197 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:35:13.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:35:13.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:13.197 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:13.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:14.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:14.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:14.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:35:14.438 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:35:14.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:14.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:14.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:15.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:15.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:15.667 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:35:15.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:35:15.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:15.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:15.897 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:16.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:16.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:16.899 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:35:16.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:35:16.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:16.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:17.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:18.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:18.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:18.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:35:18.189 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:35:18.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:18.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:18.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:19.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:19.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:19.413 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:35:19.413 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:35:19.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:19.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:19.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:20.649 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:20.650 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:20.650 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:35:20.650 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:35:20.650 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:20.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:20.877 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:21.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:21.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:21.880 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:35:21.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:35:21.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:21.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:22.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:23.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:23.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:23.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:35:23.115 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:35:23.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:23.116 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:23.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:24.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:24.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:24.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:35:24.342 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:35:24.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:24.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:24.579 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:25.580 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:25.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:25.581 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-09-30T01:35:25.581 INFO:tasks.workunit.client.0.smithi186.stdout:17 2023-09-30T01:35:25.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:25.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:25.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:26.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:26.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:26.812 INFO:tasks.workunit.client.0.smithi186.stdout:18 2023-09-30T01:35:26.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-09-30T01:35:26.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:26.813 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:27.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:28.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:28.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:28.048 INFO:tasks.workunit.client.0.smithi186.stdout:19 2023-09-30T01:35:28.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2023-09-30T01:35:28.048 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:28.049 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:28.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:29.275 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:29.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:29.276 INFO:tasks.workunit.client.0.smithi186.stdout:20 2023-09-30T01:35:29.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2023-09-30T01:35:29.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:29.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:35:29.666 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/138409,v1:127.0.0.1:6801/138409] [v2:127.0.0.1:6802/138409,v1:127.0.0.1:6803/138409] exists,up ea1bd145-b197-4f6e-83b1-e76adbed579e 2023-09-30T01:35:29.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:35:29.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:35:29.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:35:29.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: run_osd td/ceph-helpers 1 2023-09-30T01:35:29.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:35:29.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:35:29.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:35:29.669 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:35:29.670 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:35:29.670 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:35:29.670 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:35:29.670 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:35:29.670 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:35:29.671 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:35:29.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:35:29.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:35:29.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:35:29.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:35:29.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:35:29.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:35:29.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:35:29.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:35:29.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:35:29.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:35:29.672 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:35:29.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:35:29.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2023-09-30T01:35:29.673 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:35:29.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b1d1ec61-b129-4c11-85d2-d33ab3b18974 2023-09-30T01:35:29.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b1d1ec61-b129-4c11-85d2-d33ab3b18974' 2023-09-30T01:35:29.674 INFO:tasks.workunit.client.0.smithi186.stdout:add osd1 b1d1ec61-b129-4c11-85d2-d33ab3b18974 2023-09-30T01:35:29.674 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:35:29.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhexdlDTM8KRAAPWZLMxrz2Icr2D/QOkIIrw== 2023-09-30T01:35:29.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhexdlDTM8KRAAPWZLMxrz2Icr2D/QOkIIrw=="}' 2023-09-30T01:35:29.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b1d1ec61-b129-4c11-85d2-d33ab3b18974 -i td/ceph-helpers/1/new.json 2023-09-30T01:35:30.035 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:35:30.052 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2023-09-30T01:35:30.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhexdlDTM8KRAAPWZLMxrz2Icr2D/QOkIIrw== --osd-uuid b1d1ec61-b129-4c11-85d2-d33ab3b18974 2023-09-30T01:35:30.082 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:30.077+0000 7f051774e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:35:30.113 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:30.109+0000 7f051774e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:35:30.114 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:30.109+0000 7f051774e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:35:30.115 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:30.109+0000 7f051774e3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:35:30.116 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:30.113+0000 7f051774e3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:35:30.116 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:30.113+0000 7f051774e3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:35:30.159 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:30.153+0000 7f051774e3c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2023-09-30T01:35:36.141 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2023-09-30T01:35:36.141 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:35:36.143 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd1 key to auth repository 2023-09-30T01:35:36.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-09-30T01:35:36.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:35:36.384 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-09-30T01:35:36.385 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:35:36.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:35:36.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:35:36.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:35:36.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:35:36.417 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:36.413+0000 7f73f6e773c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:35:36.419 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:36.417+0000 7f73f6e773c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:35:36.423 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:36.417+0000 7f73f6e773c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:35:36.606 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-09-30T01:35:36.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:35:36.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:35:36.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:35:36.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:35:36.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:36.607 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:35:36.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:35:36.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:36.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:36.863 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:36.966 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:36.961+0000 7f73f6e773c0 -1 Falling back to public interface 2023-09-30T01:35:37.865 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:37.865 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:37.866 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:35:37.866 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:35:37.866 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:37.866 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:38.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:39.097 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:39.097 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:39.097 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:35:39.098 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:35:39.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:39.098 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:39.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:39.635 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:35:39.629+0000 7f73f6e773c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:35:40.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:40.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:40.325 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:35:40.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:35:40.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:40.326 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:40.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:41.675 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:41.675 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:41.675 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:35:41.676 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:35:41.676 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:41.676 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:41.905 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:42.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:42.907 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:42.907 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:35:42.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:35:42.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:42.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:43.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:44.138 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:44.138 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:44.138 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:35:44.139 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:35:44.139 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:44.139 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:44.365 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:45.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:45.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:45.367 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:35:45.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:35:45.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:45.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:45.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:46.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:46.598 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:46.599 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:35:46.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:35:46.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:46.599 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:46.826 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:47.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:47.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:47.829 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:35:47.829 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:35:47.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:47.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:48.055 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:49.057 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:49.057 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:49.058 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:35:49.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:35:49.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:49.058 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:49.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:50.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:50.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:50.289 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:35:50.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:35:50.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:50.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:50.515 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:51.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:51.517 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:51.517 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:35:51.518 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:35:51.518 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:51.518 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:51.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:52.753 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:52.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:52.754 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:35:52.755 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:35:52.755 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:52.755 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:52.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:53.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:53.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:53.987 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:35:53.987 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:35:53.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:53.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:54.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:55.216 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:55.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:55.217 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:35:55.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:35:55.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:55.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:55.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:56.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:56.445 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:56.446 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:35:56.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:35:56.446 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:56.447 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:56.671 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:57.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:57.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:57.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-09-30T01:35:57.674 INFO:tasks.workunit.client.0.smithi186.stdout:17 2023-09-30T01:35:57.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:57.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:57.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:35:58.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:35:58.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:35:58.910 INFO:tasks.workunit.client.0.smithi186.stdout:18 2023-09-30T01:35:58.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-09-30T01:35:58.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:35:58.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:35:59.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:00.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:00.137 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:00.137 INFO:tasks.workunit.client.0.smithi186.stdout:19 2023-09-30T01:36:00.138 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2023-09-30T01:36:00.138 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:00.138 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:36:00.365 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:01.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:01.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:01.367 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2023-09-30T01:36:01.367 INFO:tasks.workunit.client.0.smithi186.stdout:20 2023-09-30T01:36:01.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:01.368 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:36:01.723 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/140262,v1:127.0.0.1:6809/140262] [v2:127.0.0.1:6810/140262,v1:127.0.0.1:6811/140262] exists,up b1d1ec61-b129-4c11-85d2-d33ab3b18974 2023-09-30T01:36:01.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:36:01.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:36:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:36:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:506: test_run_mon: run_osd td/ceph-helpers 2 2023-09-30T01:36:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:36:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:36:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-09-30T01:36:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:36:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2023-09-30T01:36:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:36:01.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:36:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:36:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:36:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2023-09-30T01:36:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2023-09-30T01:36:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:36:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:36:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:36:01.727 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:36:01.727 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:36:01.728 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:36:01.728 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:36:01.728 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:01.728 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:01.729 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:36:01.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:36:01.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:36:01.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:36:01.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:36:01.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:36:01.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:36:01.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:36:01.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:36:01.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:36:01.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:36:01.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:36:01.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:36:01.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2023-09-30T01:36:01.731 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:36:01.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a9223510-cca6-4013-a460-ce9cdebb8895 2023-09-30T01:36:01.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a9223510-cca6-4013-a460-ce9cdebb8895' 2023-09-30T01:36:01.731 INFO:tasks.workunit.client.0.smithi186.stdout:add osd2 a9223510-cca6-4013-a460-ce9cdebb8895 2023-09-30T01:36:01.732 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:36:01.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBexdlaEWxLBAAxqkfzdPG+/BYjsJdgg+NBA== 2023-09-30T01:36:01.750 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBexdlaEWxLBAAxqkfzdPG+/BYjsJdgg+NBA=="}' 2023-09-30T01:36:01.751 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a9223510-cca6-4013-a460-ce9cdebb8895 -i td/ceph-helpers/2/new.json 2023-09-30T01:36:02.110 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:36:02.122 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2023-09-30T01:36:02.123 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBexdlaEWxLBAAxqkfzdPG+/BYjsJdgg+NBA== --osd-uuid a9223510-cca6-4013-a460-ce9cdebb8895 2023-09-30T01:36:02.150 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:02.145+0000 7fef2d9da3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:02.197 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:02.193+0000 7fef2d9da3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:02.199 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:02.193+0000 7fef2d9da3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:02.200 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:02.197+0000 7fef2d9da3c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2023-09-30T01:36:02.200 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:02.197+0000 7fef2d9da3c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2023-09-30T01:36:02.200 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:02.197+0000 7fef2d9da3c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2023-09-30T01:36:02.316 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:02.313+0000 7fef2d9da3c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2023-09-30T01:36:08.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2023-09-30T01:36:08.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:36:08.308 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd2 key to auth repository 2023-09-30T01:36:08.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-09-30T01:36:08.309 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:36:08.547 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.2 2023-09-30T01:36:08.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-09-30T01:36:08.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:36:08.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:36:08.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:36:08.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:36:08.578 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:08.572+0000 7f78a067c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:08.581 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:08.576+0000 7f78a067c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:08.584 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:08.580+0000 7f78a067c3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:08.767 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-09-30T01:36:08.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:36:08.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-09-30T01:36:08.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:36:08.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:36:08.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:08.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:36:08.768 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:36:08.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:08.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:08.996 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:09.130 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:09.124+0000 7f78a067c3c0 -1 Falling back to public interface 2023-09-30T01:36:09.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:09.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:09.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:36:09.998 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:36:09.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:09.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:10.223 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:11.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:11.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:11.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:36:11.226 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:36:11.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:11.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:11.458 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:11.733 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:11.728+0000 7f78a067c3c0 -1 osd.2 0 log_to_monitors true 2023-09-30T01:36:12.459 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:12.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:12.460 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:36:12.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:36:12.461 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:12.461 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:12.805 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:13.807 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:13.807 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:13.807 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:36:13.808 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:36:13.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:13.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:14.030 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:15.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:15.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:15.033 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:36:15.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:36:15.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:15.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:15.262 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:16.264 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:16.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:16.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:36:16.265 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:36:16.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:16.265 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:16.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:17.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:17.493 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:17.493 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:36:17.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:36:17.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:17.494 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:17.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:18.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:18.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:18.725 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:36:18.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:36:18.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:18.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:18.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:19.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:19.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:19.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:36:19.954 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:36:19.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:19.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:20.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:21.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:21.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:21.188 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:36:21.189 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:36:21.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:21.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:21.430 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:22.432 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:22.432 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:22.432 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:36:22.432 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:36:22.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:22.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:22.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:23.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:23.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:23.655 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:36:23.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:36:23.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:23.656 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:23.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:24.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:24.883 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:24.883 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:36:24.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:36:24.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:24.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:25.110 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:26.111 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:26.111 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:26.112 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:36:26.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:36:26.112 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:26.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:26.335 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:27.337 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:27.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:27.338 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:36:27.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:36:27.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:27.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:27.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:28.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:28.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:28.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:36:28.563 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:36:28.564 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:28.564 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:28.787 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:29.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:29.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:29.789 INFO:tasks.workunit.client.0.smithi186.stdout:17 2023-09-30T01:36:29.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-09-30T01:36:29.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:29.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:30.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:31.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:31.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:31.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-09-30T01:36:31.019 INFO:tasks.workunit.client.0.smithi186.stdout:18 2023-09-30T01:36:31.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:31.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:31.244 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:32.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:32.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:32.247 INFO:tasks.workunit.client.0.smithi186.stdout:19 2023-09-30T01:36:32.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2023-09-30T01:36:32.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:32.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:32.632 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:33.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:33.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:33.634 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2023-09-30T01:36:33.635 INFO:tasks.workunit.client.0.smithi186.stdout:20 2023-09-30T01:36:33.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:33.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:36:33.880 INFO:tasks.workunit.client.0.smithi186.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/142115,v1:127.0.0.1:6817/142115] [v2:127.0.0.1:6818/142115,v1:127.0.0.1:6819/142115] exists,up a9223510-cca6-4013-a460-ce9cdebb8895 2023-09-30T01:36:33.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:36:33.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:36:33.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:36:33.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:507: test_run_mon: create_rbd_pool 2023-09-30T01:36:33.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:36:34.131 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:36:34.140 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:36:34.140 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:36:34.520 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:36:34.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:36:35.531 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:36:37.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: ceph osd dump 2023-09-30T01:36:37.716 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: grep 'pool 1 '\''rbd'\''' 2023-09-30T01:36:37.942 INFO:tasks.workunit.client.0.smithi186.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 24 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2023-09-30T01:36:37.943 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: get_asok_path mon.a 2023-09-30T01:36:37.943 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:36:37.943 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:36:37.944 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:36:37.944 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:37.944 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:37.944 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:36:37.944 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: CEPH_ARGS= 2023-09-30T01:36:37.944 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get osd_pool_default_size 2023-09-30T01:36:38.021 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: local 'size={"osd_pool_default_size":"3"}' 2023-09-30T01:36:38.021 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: test '{"osd_pool_default_size":"3"}' = '{"osd_pool_default_size":"3"}' 2023-09-30T01:36:38.022 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: CEPH_ARGS= 2023-09-30T01:36:38.022 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: ceph status 2023-09-30T01:36:38.107 INFO:tasks.workunit.client.0.smithi186.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2023-09-30T01:36:38.107 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.100+0000 7f8ce65f7700 -1 failed for service _ceph-mon._tcp 2023-09-30T01:36:38.108 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.100+0000 7f8ce65f7700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-09-30T01:36:38.108 INFO:tasks.workunit.client.0.smithi186.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2023-09-30T01:36:38.108 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: CEPH_ARGS= 2023-09-30T01:36:38.108 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: ceph --conf td/ceph-helpers/ceph.conf status 2023-09-30T01:36:38.189 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.184+0000 7fa6b9cdc700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2023-09-30T01:36:38.189 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.184+0000 7fa6b9cdc700 -1 AuthRegistry(0x7fa6b40646d0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2023-09-30T01:36:38.189 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.184+0000 7fa6b9cdc700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2023-09-30T01:36:38.189 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.184+0000 7fa6b9cdc700 -1 AuthRegistry(0x7fa6b9cdafe0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2023-09-30T01:36:38.193 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.188+0000 7fa6b9cdc700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2023-09-30T01:36:38.193 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.188+0000 7fa6b9cdc700 -1 AuthRegistry(0x7fa6b40615f0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2023-09-30T01:36:38.193 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.188+0000 7fa6b9cdc700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2023-09-30T01:36:38.193 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.188+0000 7fa6b9cdc700 -1 AuthRegistry(0x7fa6b4100410) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2023-09-30T01:36:38.193 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.188+0000 7fa6b9cdc700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2023-09-30T01:36:38.193 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:38.188+0000 7fa6b9cdc700 -1 AuthRegistry(0x7fa6b40615f0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2023-09-30T01:36:38.306 INFO:tasks.workunit.client.0.smithi186.stdout: cluster: 2023-09-30T01:36:38.306 INFO:tasks.workunit.client.0.smithi186.stdout: id: 779146aa-b0f9-4583-8ddd-7df6da70ad9d 2023-09-30T01:36:38.306 INFO:tasks.workunit.client.0.smithi186.stdout: health: HEALTH_OK 2023-09-30T01:36:38.306 INFO:tasks.workunit.client.0.smithi186.stdout: 2023-09-30T01:36:38.307 INFO:tasks.workunit.client.0.smithi186.stdout: services: 2023-09-30T01:36:38.307 INFO:tasks.workunit.client.0.smithi186.stdout: mon: 1 daemons, quorum a (age 101s) 2023-09-30T01:36:38.307 INFO:tasks.workunit.client.0.smithi186.stdout: mgr: no daemons active 2023-09-30T01:36:38.307 INFO:tasks.workunit.client.0.smithi186.stdout: osd: 3 osds: 3 up (since 5s), 3 in (since 36s) 2023-09-30T01:36:38.307 INFO:tasks.workunit.client.0.smithi186.stdout: 2023-09-30T01:36:38.307 INFO:tasks.workunit.client.0.smithi186.stdout: data: 2023-09-30T01:36:38.307 INFO:tasks.workunit.client.0.smithi186.stdout: pools: 0 pools, 0 pgs 2023-09-30T01:36:38.307 INFO:tasks.workunit.client.0.smithi186.stdout: objects: 0 objects, 0 B 2023-09-30T01:36:38.307 INFO:tasks.workunit.client.0.smithi186.stdout: usage: 0 B used, 0 B / 0 B avail 2023-09-30T01:36:38.307 INFO:tasks.workunit.client.0.smithi186.stdout: pgs: 2023-09-30T01:36:38.307 INFO:tasks.workunit.client.0.smithi186.stdout: 2023-09-30T01:36:38.316 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:516: test_run_mon: kill_daemons td/ceph-helpers 2023-09-30T01:36:38.317 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:36:38.317 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:36:38.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:36:38.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:36:38.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:36:43.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:36:43.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:36:43.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:36:43.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:36:43.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:36:43.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:36:43.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:36:43.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:36:43.668 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:43.664+0000 7fa69abdba00 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2023-09-30T01:36:43.670 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:36:43.671 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:36:43.671 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:36:43.671 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:36:43.671 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:43.671 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:43.672 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:36:43.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:36:43.981 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:36:43.983 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:36:43.983 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:36:43.983 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:36:43.983 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:36:43.985 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:36:43.985 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:36:43.985 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:36:43.985 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:36:43.985 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:36:43.985 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:43.985 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:43.985 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:36:43.986 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:36:43.986 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:36:44.072 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:36:44.072 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:36:44.072 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:36:44.073 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:36:44.073 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:36:44.073 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:36:44.074 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:36:44.074 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:36:44.074 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:36:44.074 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:44.074 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:44.075 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:36:44.075 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:36:44.075 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:36:44.165 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: get_asok_path mon.a 2023-09-30T01:36:44.165 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:36:44.166 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:36:44.166 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:36:44.166 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:44.166 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:44.166 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:36:44.166 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: CEPH_ARGS= 2023-09-30T01:36:44.166 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get osd_pool_default_size 2023-09-30T01:36:44.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: local 'size={"osd_pool_default_size":"1"}' 2023-09-30T01:36:44.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:521: test_run_mon: test '{"osd_pool_default_size":"1"}' = '{"osd_pool_default_size":"1"}' 2023-09-30T01:36:44.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:522: test_run_mon: kill_daemons td/ceph-helpers 2023-09-30T01:36:44.246 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:36:44.246 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:36:44.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:36:44.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:36:44.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:36:44.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:36:44.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: CEPH_ARGS='--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd_pool_default_size=2' 2023-09-30T01:36:44.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: run_mon td/ceph-helpers a 2023-09-30T01:36:44.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:36:44.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:36:44.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:36:44.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:36:44.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:36:44.359 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2023-09-30T01:36:44.387 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:44.380+0000 7f112cc02a00 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2023-09-30T01:36:44.389 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:36:44.390 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:36:44.390 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:36:44.390 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:36:44.390 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:44.390 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:44.391 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:36:44.391 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:36:44.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:36:44.732 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:36:44.732 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:36:44.732 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:36:44.732 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:36:44.733 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:36:44.733 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:36:44.734 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:36:44.734 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:36:44.735 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:36:44.735 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:44.735 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:44.736 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:36:44.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:36:44.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:36:44.832 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:36:44.832 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:36:44.832 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:36:44.833 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:36:44.833 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:36:44.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:36:44.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:36:44.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:36:44.834 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:36:44.834 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:44.834 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:44.834 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:36:44.835 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:36:44.835 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:36:44.922 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: get_asok_path mon.a 2023-09-30T01:36:44.922 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:36:44.922 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:36:44.923 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:36:44.923 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:44.923 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:44.923 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:36:44.923 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: CEPH_ARGS= 2023-09-30T01:36:44.924 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get osd_pool_default_size 2023-09-30T01:36:45.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: local 'size={"osd_pool_default_size":"2"}' 2023-09-30T01:36:45.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: test_run_mon: test '{"osd_pool_default_size":"2"}' = '{"osd_pool_default_size":"2"}' 2023-09-30T01:36:45.008 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:529: test_run_mon: kill_daemons td/ceph-helpers 2023-09-30T01:36:45.008 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:36:45.008 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:36:45.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:36:45.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:36:45.009 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:36:45.123 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:36:45.123 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:531: test_run_mon: teardown td/ceph-helpers 2023-09-30T01:36:45.123 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:36:45.123 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:36:45.123 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:36:45.124 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:36:45.124 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:36:45.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:36:45.125 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:36:45.125 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:36:45.136 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:36:45.137 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:36:45.138 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:36:45.138 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:36:45.140 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:36:45.140 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:36:45.140 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:36:45.141 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:36:45.141 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:36:45.142 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:36:45.143 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:36:45.144 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:36:45.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:36:45.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:36:45.162 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:36:45.163 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:45.163 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:45.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:36:45.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:36:45.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:36:45.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:36:45.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_run_osd td/ceph-helpers 2023-09-30T01:36:45.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: local dir=td/ceph-helpers 2023-09-30T01:36:45.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:747: test_run_osd: setup td/ceph-helpers 2023-09-30T01:36:45.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:36:45.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:36:45.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:36:45.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:36:45.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:36:45.166 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:36:45.166 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:36:45.166 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:36:45.166 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:36:45.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:36:45.170 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:36:45.170 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:36:45.171 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:36:45.172 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:36:45.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:36:45.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:36:45.174 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:36:45.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:36:45.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:36:45.176 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:36:45.176 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:36:45.178 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:36:45.179 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:36:45.179 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:36:45.181 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:36:45.181 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:45.181 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:45.182 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:36:45.183 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:36:45.183 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:36:45.183 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:36:45.186 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:36:45.186 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:45.186 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:45.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:36:45.188 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:36:45.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:36:45.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:36:45.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:36:45.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:749: test_run_osd: run_mon td/ceph-helpers a 2023-09-30T01:36:45.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:36:45.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:36:45.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:36:45.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:36:45.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:36:45.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2023-09-30T01:36:45.572 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:36:45.572 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:36:45.572 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:36:45.573 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:36:45.573 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:45.573 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:45.573 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:36:45.574 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2023-09-30T01:36:45.885 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:36:45.887 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:36:45.887 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:36:45.887 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:36:45.887 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:36:45.887 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:36:45.888 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:36:45.888 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:36:45.888 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:36:45.889 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:36:45.889 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:45.889 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:45.889 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:36:45.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:36:45.890 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:36:45.980 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:36:45.980 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:36:45.980 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:36:45.981 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:36:45.981 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:36:45.982 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:36:45.982 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:36:45.982 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:36:45.982 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:36:45.982 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:45.982 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:45.983 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:36:45.983 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:36:45.983 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:36:46.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: run_mgr td/ceph-helpers x 2023-09-30T01:36:46.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:36:46.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:36:46.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:36:46.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:36:46.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:36:46.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:36:46.962 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:36:46.962 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:36:46.962 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:36:46.963 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:36:46.963 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:46.963 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:46.963 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:36:46.963 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:36:46.964 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:36:47.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:752: test_run_osd: run_osd td/ceph-helpers 0 2023-09-30T01:36:47.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:36:47.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:36:47.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:36:47.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:36:47.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:36:47.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:36:47.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:36:47.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:36:47.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:36:47.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:36:47.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:36:47.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:36:47.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:36:47.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:36:47.001 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:36:47.001 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:36:47.001 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:36:47.003 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:36:47.003 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:36:47.003 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:36:47.003 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:36:47.003 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:36:47.003 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:36:47.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:36:47.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:36:47.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:36:47.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:36:47.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:36:47.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:36:47.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:36:47.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:36:47.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:36:47.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:36:47.004 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:36:47.005 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:36:47.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d38efcb2-9d7e-4c6e-b7fb-8ad0b8a1650a 2023-09-30T01:36:47.006 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d38efcb2-9d7e-4c6e-b7fb-8ad0b8a1650a' 2023-09-30T01:36:47.006 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 d38efcb2-9d7e-4c6e-b7fb-8ad0b8a1650a 2023-09-30T01:36:47.007 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:36:47.024 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvexdlvD9sARAAV172pmAzUIrHJ1vVm2bevw== 2023-09-30T01:36:47.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvexdlvD9sARAAV172pmAzUIrHJ1vVm2bevw=="}' 2023-09-30T01:36:47.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d38efcb2-9d7e-4c6e-b7fb-8ad0b8a1650a -i td/ceph-helpers/0/new.json 2023-09-30T01:36:47.414 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:36:47.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:36:47.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvexdlvD9sARAAV172pmAzUIrHJ1vVm2bevw== --osd-uuid d38efcb2-9d7e-4c6e-b7fb-8ad0b8a1650a 2023-09-30T01:36:47.463 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:47.456+0000 7f74799313c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:47.466 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:47.460+0000 7f74799313c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:47.467 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:47.460+0000 7f74799313c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:47.467 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:47.464+0000 7f74799313c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:36:47.468 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:47.464+0000 7f74799313c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:36:47.468 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:47.464+0000 7f74799313c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:36:47.536 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:47.532+0000 7f74799313c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:36:54.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:36:54.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:36:54.135 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:36:54.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:36:54.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:36:54.689 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:36:54.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:36:54.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:36:54.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:36:54.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:36:54.690 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:36:54.719 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:54.712+0000 7fe124b3d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:54.721 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:54.716+0000 7fe124b3d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:54.723 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:54.716+0000 7fe124b3d3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:36:55.002 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:54.996+0000 7fe124b3d3c0 -1 Falling back to public interface 2023-09-30T01:36:55.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:36:55.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:36:55.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:36:55.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:36:55.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:36:55.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:55.209 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:36:55.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:36:55.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:55.209 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:36:55.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:56.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:56.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:56.699 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:36:56.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:36:56.699 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:56.700 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:36:57.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:57.712 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:36:57.708+0000 7fe124b3d3c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:36:58.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:58.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:58.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:36:58.095 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:36:58.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:58.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:36:58.501 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:36:59.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:36:59.503 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:36:59.504 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:36:59.504 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:36:59.504 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:36:59.504 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:36:59.994 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:00.996 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:00.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:00.997 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:37:00.997 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:37:00.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:00.998 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:01.470 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:02.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:02.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:02.472 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:37:02.473 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:37:02.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:02.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:02.915 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:03.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:03.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:03.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:37:03.917 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:37:03.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:03.918 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:04.521 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:05.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:05.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:05.523 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:37:05.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:37:05.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:05.524 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:05.935 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:06.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:06.937 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:06.937 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:37:06.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:37:06.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:06.938 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:07.435 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:08.436 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:08.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:08.437 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:37:08.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:37:08.437 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:08.438 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:08.847 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:09.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:09.850 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:09.850 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:37:09.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:37:09.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:09.851 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:10.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:11.351 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:11.351 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:11.352 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:37:11.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:37:11.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:11.352 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:11.877 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:12.878 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:12.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:12.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:37:12.879 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:37:12.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:12.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:13.809 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:14.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:14.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:14.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:37:14.812 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:37:14.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:14.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:15.397 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:16.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:16.399 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:16.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:37:16.400 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:37:16.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:16.400 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:17.355 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:18.357 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:18.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:18.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:37:18.358 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:37:18.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:18.358 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:18.908 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:19.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:19.909 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:19.910 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:37:19.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:37:19.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:19.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:20.488 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:21.490 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:21.491 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:21.491 INFO:tasks.workunit.client.0.smithi186.stdout:17 2023-09-30T01:37:21.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-09-30T01:37:21.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:21.492 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:37:21.947 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/144592,v1:127.0.0.1:6803/144592] [v2:127.0.0.1:6804/144592,v1:127.0.0.1:6805/144592] exists,up d38efcb2-9d7e-4c6e-b7fb-8ad0b8a1650a 2023-09-30T01:37:21.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:37:21.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:37:21.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:37:21.949 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: get_asok_path osd.0 2023-09-30T01:37:21.949 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2023-09-30T01:37:21.949 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2023-09-30T01:37:21.950 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:37:21.950 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:37:21.950 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:37:21.950 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.0.asok 2023-09-30T01:37:21.951 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: CEPH_ARGS= 2023-09-30T01:37:21.951 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-osd.0.asok config get osd_max_backfills 2023-09-30T01:37:22.035 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: local 'backfills={"osd_max_backfills":"1"}' 2023-09-30T01:37:22.036 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: echo '{"osd_max_backfills":"1"}' 2023-09-30T01:37:22.036 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: grep --quiet osd_max_backfills 2023-09-30T01:37:22.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:757: test_run_osd: run_osd td/ceph-helpers 1 --osd-max-backfills 20 2023-09-30T01:37:22.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:37:22.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:37:22.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-09-30T01:37:22.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:37:22.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2023-09-30T01:37:22.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:37:22.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:37:22.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:37:22.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:37:22.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2023-09-30T01:37:22.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2023-09-30T01:37:22.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:37:22.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:37:22.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:37:22.043 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:37:22.043 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:37:22.043 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:37:22.043 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:37:22.043 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:37:22.043 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:37:22.044 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:37:22.044 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:37:22.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:37:22.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:37:22.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:37:22.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:37:22.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:37:22.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:37:22.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:37:22.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:37:22.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:37:22.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:37:22.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-max-backfills 20' 2023-09-30T01:37:22.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2023-09-30T01:37:22.046 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:37:22.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bbb14b3b-0a69-43bf-9411-7dadbd76e5ed 2023-09-30T01:37:22.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bbb14b3b-0a69-43bf-9411-7dadbd76e5ed' 2023-09-30T01:37:22.047 INFO:tasks.workunit.client.0.smithi186.stdout:add osd1 bbb14b3b-0a69-43bf-9411-7dadbd76e5ed 2023-09-30T01:37:22.047 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:37:22.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSexdlolvhAxAAcn1LCXY8Q5d57sAhCcJGmA== 2023-09-30T01:37:22.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSexdlolvhAxAAcn1LCXY8Q5d57sAhCcJGmA=="}' 2023-09-30T01:37:22.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bbb14b3b-0a69-43bf-9411-7dadbd76e5ed -i td/ceph-helpers/1/new.json 2023-09-30T01:37:22.594 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:37:22.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2023-09-30T01:37:22.609 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 --mkfs --key AQDSexdlolvhAxAAcn1LCXY8Q5d57sAhCcJGmA== --osd-uuid bbb14b3b-0a69-43bf-9411-7dadbd76e5ed 2023-09-30T01:37:22.643 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:22.639+0000 7f76cb1133c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:37:22.646 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:22.643+0000 7f76cb1133c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:37:22.648 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:22.647+0000 7f76cb1133c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:37:22.649 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:22.647+0000 7f76cb1133c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:37:22.649 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:22.647+0000 7f76cb1133c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:37:22.650 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:22.647+0000 7f76cb1133c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:37:22.749 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:22.747+0000 7f76cb1133c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2023-09-30T01:37:28.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2023-09-30T01:37:28.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:37:28.843 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd1 key to auth repository 2023-09-30T01:37:28.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-09-30T01:37:28.844 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:37:29.413 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:37:29.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-09-30T01:37:29.414 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:37:29.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:37:29.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2023-09-30T01:37:29.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:37:29.443 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:29.439+0000 7f2536bb53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:37:29.445 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:29.443+0000 7f2536bb53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:37:29.447 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:29.443+0000 7f2536bb53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:37:29.777 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-09-30T01:37:29.777 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:37:29.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:37:29.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:37:29.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:37:29.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:29.778 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:37:29.778 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:37:29.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:29.779 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:29.998 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:29.995+0000 7f2536bb53c0 -1 Falling back to public interface 2023-09-30T01:37:30.208 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:31.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:31.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:31.210 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:37:31.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:37:31.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:31.211 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:31.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:32.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:32.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:32.615 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:37:32.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:37:32.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:32.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:32.663 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:32.659+0000 7f2536bb53c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:37:33.050 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:34.052 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:34.052 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:34.053 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:37:34.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:37:34.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:34.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:34.616 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:35.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:35.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:35.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:37:35.619 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:37:35.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:35.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:36.028 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:37.030 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:37.030 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:37.030 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:37:37.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:37:37.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:37.031 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:37.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:38.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:38.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:38.566 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:37:38.566 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:37:38.567 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:38.567 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:39.066 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:40.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:40.068 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:40.069 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:37:40.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:37:40.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:40.069 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:40.551 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:41.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:41.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:41.554 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:37:41.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:37:41.554 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:41.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:41.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:42.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:42.954 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:42.954 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:37:42.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:37:42.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:42.955 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:43.499 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:44.501 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:44.501 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:44.501 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:37:44.502 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:37:44.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:44.502 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:45.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:46.462 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:46.462 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:46.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:37:46.463 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:37:46.463 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:46.464 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:46.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:48.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:48.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:48.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:37:48.001 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:37:48.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:48.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:48.944 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:49.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:49.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:49.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:37:49.947 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:37:49.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:49.947 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:50.825 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:51.827 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:51.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:51.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:37:51.828 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:37:51.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:51.828 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:52.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:53.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:53.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:53.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:37:53.389 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:37:53.390 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:53.390 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:54.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:37:55.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:37:55.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:37:55.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:37:55.237 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:37:55.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:37:55.238 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:37:55.639 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/146373,v1:127.0.0.1:6811/146373] [v2:127.0.0.1:6812/146373,v1:127.0.0.1:6813/146373] exists,up bbb14b3b-0a69-43bf-9411-7dadbd76e5ed 2023-09-30T01:37:55.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:37:55.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:37:55.640 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:37:55.641 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: get_asok_path osd.1 2023-09-30T01:37:55.641 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-09-30T01:37:55.641 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-09-30T01:37:55.642 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:37:55.642 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:37:55.642 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:37:55.642 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.1.asok 2023-09-30T01:37:55.643 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: CEPH_ARGS= 2023-09-30T01:37:55.643 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-osd.1.asok config get osd_max_backfills 2023-09-30T01:37:55.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: local 'backfills={"osd_max_backfills":"20"}' 2023-09-30T01:37:55.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:760: test_run_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2023-09-30T01:37:55.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: CEPH_ARGS='--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2023-09-30T01:37:55.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: run_osd td/ceph-helpers 2 2023-09-30T01:37:55.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:37:55.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:37:55.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-09-30T01:37:55.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:37:55.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2023-09-30T01:37:55.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2023-09-30T01:37:55.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:37:55.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:37:55.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:37:55.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2023-09-30T01:37:55.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2023-09-30T01:37:55.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:37:55.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:37:55.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:37:55.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:37:55.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:37:55.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:37:55.735 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:37:55.735 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:37:55.735 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:37:55.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:37:55.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:37:55.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:37:55.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:37:55.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:37:55.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:37:55.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:37:55.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:37:55.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:37:55.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:37:55.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:37:55.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:37:55.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:37:55.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2023-09-30T01:37:55.737 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:37:55.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd27a6d4-565d-4474-95ea-7c48a08db1b6 2023-09-30T01:37:55.738 INFO:tasks.workunit.client.0.smithi186.stdout:add osd2 cd27a6d4-565d-4474-95ea-7c48a08db1b6 2023-09-30T01:37:55.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cd27a6d4-565d-4474-95ea-7c48a08db1b6' 2023-09-30T01:37:55.739 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:37:55.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzexdlhwEQLRAAhfMx8+QWE6zA8jfnW27xrA== 2023-09-30T01:37:55.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzexdlhwEQLRAAhfMx8+QWE6zA8jfnW27xrA=="}' 2023-09-30T01:37:55.757 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd27a6d4-565d-4474-95ea-7c48a08db1b6 -i td/ceph-helpers/2/new.json 2023-09-30T01:37:56.408 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:37:56.422 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2023-09-30T01:37:56.424 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzexdlhwEQLRAAhfMx8+QWE6zA8jfnW27xrA== --osd-uuid cd27a6d4-565d-4474-95ea-7c48a08db1b6 2023-09-30T01:37:56.453 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:56.451+0000 7f496a7c03c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:37:56.455 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:56.451+0000 7f496a7c03c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:37:56.457 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:56.455+0000 7f496a7c03c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:37:56.458 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:56.455+0000 7f496a7c03c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2023-09-30T01:37:56.458 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:56.455+0000 7f496a7c03c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2023-09-30T01:37:56.459 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:56.455+0000 7f496a7c03c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2023-09-30T01:37:56.521 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:37:56.519+0000 7f496a7c03c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2023-09-30T01:38:02.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2023-09-30T01:38:02.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:38:02.864 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd2 key to auth repository 2023-09-30T01:38:02.864 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-09-30T01:38:02.864 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:38:03.400 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.2 2023-09-30T01:38:03.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-09-30T01:38:03.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:38:03.401 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:38:03.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:38:03.402 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:38:03.433 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:03.430+0000 7f2aca42a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:38:03.435 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:03.430+0000 7f2aca42a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:38:03.436 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:03.434+0000 7f2aca42a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:38:03.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-09-30T01:38:03.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:38:03.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-09-30T01:38:03.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:38:03.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:38:03.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:03.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:38:03.840 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:38:03.840 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:03.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:03.986 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:03.982+0000 7f2aca42a3c0 -1 Falling back to public interface 2023-09-30T01:38:04.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:05.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:05.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:05.250 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:38:05.251 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:38:05.251 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:05.251 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:05.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:06.646 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:06.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:06.647 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:38:06.647 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:38:06.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:06.648 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:06.685 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:06.682+0000 7f2aca42a3c0 -1 osd.2 0 log_to_monitors true 2023-09-30T01:38:07.183 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:08.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:08.185 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:08.185 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:38:08.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:38:08.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:08.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:08.664 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:09.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:09.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:09.666 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:38:09.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:38:09.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:09.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:10.086 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:11.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:11.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:11.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:38:11.089 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:38:11.089 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:11.089 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:11.624 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:12.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:12.626 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:12.626 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:38:12.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:38:12.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:12.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:13.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:14.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:14.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:14.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:38:14.065 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:38:14.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:14.065 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:14.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:15.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:15.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:15.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:38:15.562 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:38:15.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:15.563 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:16.038 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:17.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:17.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:17.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:38:17.041 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:38:17.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:17.041 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:17.480 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:18.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:18.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:18.482 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:38:18.483 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:38:18.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:18.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:19.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:20.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:20.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:20.618 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:38:20.618 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:38:20.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:20.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:21.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:22.545 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:22.546 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:22.546 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:38:22.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:38:22.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:22.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:23.099 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:24.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:24.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:24.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:38:24.101 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:38:24.101 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:24.102 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:24.978 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:25.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:25.982 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:25.982 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:38:25.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:38:25.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:25.983 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:26.884 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:27.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:27.886 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:27.886 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:38:27.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:38:27.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:27.887 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:28.970 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:29.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:29.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:29.973 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:38:29.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:38:29.973 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:29.974 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:38:30.452 INFO:tasks.workunit.client.0.smithi186.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/148126,v1:127.0.0.1:6819/148126] [v2:127.0.0.1:6820/148126,v1:127.0.0.1:6821/148126] exists,up cd27a6d4-565d-4474-95ea-7c48a08db1b6 2023-09-30T01:38:30.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:38:30.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:38:30.453 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:38:30.454 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: get_asok_path osd.2 2023-09-30T01:38:30.455 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2023-09-30T01:38:30.455 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2023-09-30T01:38:30.455 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:38:30.455 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:30.456 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:30.456 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-osd.2.asok 2023-09-30T01:38:30.457 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: CEPH_ARGS= 2023-09-30T01:38:30.457 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.76804/ceph-osd.2.asok config get osd_max_backfills 2023-09-30T01:38:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: local 'backfills={"osd_max_backfills":"30"}' 2023-09-30T01:38:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:765: test_run_osd: test '{"osd_max_backfills":"30"}' = '{"osd_max_backfills":"30"}' 2023-09-30T01:38:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:767: test_run_osd: teardown td/ceph-helpers 2023-09-30T01:38:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:38:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:38:30.547 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:38:30.548 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:38:30.548 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:38:30.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:38:30.548 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:38:30.549 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:38:30.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:38:30.664 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:38:30.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:38:30.666 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:38:30.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:38:30.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:38:30.669 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:38:30.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:30.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:38:30.671 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:38:30.672 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:30.673 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:30.676 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:38:30.676 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:38:30.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:38:30.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:30.693 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:30.693 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:38:30.695 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:38:30.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:38:30.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:38:30.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_set_config td/ceph-helpers 2023-09-30T01:38:30.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: test_set_config: local dir=td/ceph-helpers 2023-09-30T01:38:30.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: test_set_config: setup td/ceph-helpers 2023-09-30T01:38:30.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:38:30.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:38:30.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:38:30.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:38:30.696 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:38:30.697 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:38:30.697 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:38:30.697 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:38:30.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:38:30.698 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:38:30.701 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:38:30.701 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:38:30.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:38:30.703 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:38:30.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:38:30.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:38:30.705 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:38:30.706 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:30.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:38:30.707 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:38:30.708 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:30.709 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:30.711 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:38:30.711 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:38:30.713 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:38:30.713 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:30.713 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:30.713 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:38:30.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:38:30.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:38:30.715 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:38:30.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:38:30.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:30.717 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:30.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:38:30.720 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:38:30.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:38:30.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:38:30.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:38:30.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: test_set_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:38:30.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:38:30.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:38:30.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:38:30.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:38:30.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:38:30.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:38:31.105 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:38:31.106 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:38:31.106 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:38:31.106 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:38:31.107 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:31.107 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:31.107 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:38:31.107 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-09-30T01:38:31.412 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:38:31.414 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:38:31.414 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:38:31.414 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:38:31.415 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:38:31.415 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:38:31.416 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:38:31.416 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:38:31.416 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:38:31.416 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:38:31.417 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:31.417 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:31.417 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:38:31.417 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:38:31.418 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:38:31.518 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:38:31.518 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:38:31.519 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:38:31.519 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:38:31.519 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:38:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:38:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:38:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:38:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:38:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:31.519 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:31.520 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:38:31.520 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:38:31.520 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:38:31.607 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: get_config mon a ms_crc_header 2023-09-30T01:38:31.607 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:38:31.607 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:38:31.608 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2023-09-30T01:38:31.609 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:38:31.609 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2023-09-30T01:38:31.609 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:38:31.609 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:38:31.610 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:38:31.610 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:31.610 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:31.610 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:38:31.610 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:38:31.611 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get ms_crc_header 2023-09-30T01:38:31.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: test true = true 2023-09-30T01:38:31.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: test_set_config: set_config mon a ms_crc_header false 2023-09-30T01:38:31.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2023-09-30T01:38:31.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2023-09-30T01:38:31.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2023-09-30T01:38:31.691 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=false 2023-09-30T01:38:31.692 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2023-09-30T01:38:31.692 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2023-09-30T01:38:31.692 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:38:31.692 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:38:31.693 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:38:31.693 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:31.693 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:31.693 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:38:31.694 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config set ms_crc_header false 2023-09-30T01:38:31.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2023-09-30T01:38:31.777 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: get_config mon a ms_crc_header 2023-09-30T01:38:31.777 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:38:31.778 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:38:31.778 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2023-09-30T01:38:31.779 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2023-09-30T01:38:31.779 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:38:31.779 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:38:31.780 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:38:31.780 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:38:31.780 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:31.780 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:31.781 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:38:31.782 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:38:31.782 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get ms_crc_header 2023-09-30T01:38:31.878 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: test false = false 2023-09-30T01:38:31.878 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1179: test_set_config: set_config mon a ms_crc_header true 2023-09-30T01:38:31.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2023-09-30T01:38:31.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2023-09-30T01:38:31.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2023-09-30T01:38:31.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2023-09-30T01:38:31.879 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2023-09-30T01:38:31.881 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2023-09-30T01:38:31.881 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:38:31.881 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:38:31.881 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:38:31.881 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:31.881 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:31.881 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:38:31.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config set ms_crc_header true 2023-09-30T01:38:31.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2023-09-30T01:38:31.962 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: get_config mon a ms_crc_header 2023-09-30T01:38:31.963 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:38:31.963 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:38:31.963 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2023-09-30T01:38:31.963 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2023-09-30T01:38:31.965 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:38:31.965 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:38:31.965 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:38:31.965 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:38:31.965 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:31.965 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:31.965 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:38:31.966 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:38:31.966 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get ms_crc_header 2023-09-30T01:38:32.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: test true = true 2023-09-30T01:38:32.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1181: test_set_config: teardown td/ceph-helpers 2023-09-30T01:38:32.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:38:32.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:38:32.045 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:38:32.046 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:38:32.046 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:38:32.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:38:32.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:38:32.046 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:38:32.156 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:38:32.157 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:38:32.158 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:38:32.159 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:38:32.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:38:32.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:38:32.162 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:38:32.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:38:32.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:38:32.165 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.166 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:32.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:38:32.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:38:32.170 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:38:32.171 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:32.171 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:32.171 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:38:32.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:38:32.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:38:32.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:38:32.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_setup td/ceph-helpers 2023-09-30T01:38:32.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:144: test_setup: local dir=td/ceph-helpers 2023-09-30T01:38:32.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:145: test_setup: setup td/ceph-helpers 2023-09-30T01:38:32.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:38:32.173 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:38:32.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:38:32.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:38:32.174 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:38:32.174 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:38:32.175 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:38:32.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:38:32.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:38:32.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:38:32.178 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:38:32.179 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:38:32.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:38:32.181 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:38:32.182 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:38:32.182 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:38:32.183 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:38:32.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:38:32.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:38:32.185 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:32.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:38:32.189 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:38:32.191 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:38:32.191 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:32.191 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:32.191 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:38:32.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:38:32.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:38:32.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:38:32.196 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:38:32.196 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:32.196 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:32.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:38:32.199 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:38:32.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:38:32.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:38:32.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:38:32.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:146: test_setup: test -d td/ceph-helpers 2023-09-30T01:38:32.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:147: test_setup: setup td/ceph-helpers 2023-09-30T01:38:32.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:38:32.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:38:32.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:38:32.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:38:32.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:38:32.201 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:38:32.202 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:38:32.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:38:32.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:38:32.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:38:32.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:38:32.206 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:38:32.207 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:38:32.208 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:38:32.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:38:32.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:38:32.211 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:38:32.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:38:32.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:38:32.213 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.215 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:32.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:38:32.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:38:32.219 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:38:32.220 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:32.220 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:32.220 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:38:32.221 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:38:32.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:38:32.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:38:32.224 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:38:32.224 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:32.224 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:32.225 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:38:32.227 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:38:32.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:38:32.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:38:32.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:38:32.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:148: test_setup: test -d td/ceph-helpers 2023-09-30T01:38:32.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:149: test_setup: teardown td/ceph-helpers 2023-09-30T01:38:32.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:38:32.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:38:32.229 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:38:32.230 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:38:32.230 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:38:32.230 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:38:32.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:38:32.231 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:38:32.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:38:32.236 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:38:32.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:38:32.238 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:38:32.240 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:38:32.240 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:38:32.241 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:38:32.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:38:32.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:38:32.244 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:32.248 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:38:32.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:38:32.250 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:38:32.250 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:32.250 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:32.250 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:38:32.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:38:32.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:38:32.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:38:32.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_teardown td/ceph-helpers 2023-09-30T01:38:32.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:220: test_teardown: local dir=td/ceph-helpers 2023-09-30T01:38:32.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221: test_teardown: setup td/ceph-helpers 2023-09-30T01:38:32.253 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:38:32.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:38:32.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:38:32.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:38:32.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:38:32.255 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:38:32.255 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:38:32.255 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:38:32.255 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:38:32.255 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:38:32.258 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:38:32.259 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:38:32.261 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:38:32.261 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:38:32.264 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:38:32.264 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:38:32.264 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:38:32.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:38:32.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:38:32.267 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:32.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:38:32.270 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:38:32.272 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:38:32.272 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:32.272 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:32.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:38:32.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:38:32.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:38:32.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:38:32.276 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:38:32.276 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:32.276 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:32.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:38:32.278 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:38:32.279 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:38:32.279 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:38:32.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:38:32.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222: test_teardown: teardown td/ceph-helpers 2023-09-30T01:38:32.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:38:32.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:38:32.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:38:32.281 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:38:32.281 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:38:32.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:38:32.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:38:32.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:38:32.285 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:38:32.285 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:38:32.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:38:32.288 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:38:32.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:38:32.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:38:32.290 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:38:32.291 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.291 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:38:32.292 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:38:32.293 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:32.294 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:32.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:38:32.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:38:32.299 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:38:32.299 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:32.299 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:32.299 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:38:32.301 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:38:32.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:38:32.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223: test_teardown: test -d td/ceph-helpers 2023-09-30T01:38:32.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:38:32.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_wait_background td/ceph-helpers 2023-09-30T01:38:32.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: test_wait_background: local pids= 2023-09-30T01:38:32.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 1' 2023-09-30T01:38:32.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2023-09-30T01:38:32.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2023-09-30T01:38:32.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 149294"' 2023-09-30T01:38:32.302 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 149294' 2023-09-30T01:38:32.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2023-09-30T01:38:32.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2023-09-30T01:38:32.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2023-09-30T01:38:32.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 149296"' 2023-09-30T01:38:32.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: bash -c 'sleep 1; exit 1' 2023-09-30T01:38:32.305 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 149296' 2023-09-30T01:38:32.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: test_wait_background: wait_background pids 2023-09-30T01:38:32.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: wait_background: pids=' 149294 149296' 2023-09-30T01:38:32.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_background: return_code=0 2023-09-30T01:38:32.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/149297: /' 2023-09-30T01:38:32.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: wait_background: for pid in $pids 2023-09-30T01:38:32.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: wait_background: wait 149294 2023-09-30T01:38:32.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: bash -c 'sleep 2; exit 0' 2023-09-30T01:38:32.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/149299: /' 2023-09-30T01:38:33.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 1 2023-09-30T01:38:33.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2101: wait_background: return_code=1 2023-09-30T01:38:33.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: wait_background: for pid in $pids 2023-09-30T01:38:33.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: wait_background: wait 149296 2023-09-30T01:38:34.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 0 2023-09-30T01:38:34.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: wait_background: eval 'pids='\'''\''' 2023-09-30T01:38:34.310 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: wait_background: pids= 2023-09-30T01:38:34.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: wait_background: return 1 2023-09-30T01:38:34.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: test_wait_background: '[' 1 -ne 1 ']' 2023-09-30T01:38:34.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2119: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 0' 2023-09-30T01:38:34.310 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2023-09-30T01:38:34.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2023-09-30T01:38:34.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 149302"' 2023-09-30T01:38:34.311 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 149302' 2023-09-30T01:38:34.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2023-09-30T01:38:34.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: run_in_background: local pid_variable=pids 2023-09-30T01:38:34.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: run_in_background: shift 2023-09-30T01:38:34.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: eval 'pids+=" 149304"' 2023-09-30T01:38:34.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: bash -c 'sleep 1; exit 0' 2023-09-30T01:38:34.311 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: run_in_background: pids+=' 149304' 2023-09-30T01:38:34.311 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: test_wait_background: wait_background pids 2023-09-30T01:38:34.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2095: wait_background: pids=' 149302 149304' 2023-09-30T01:38:34.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_background: return_code=0 2023-09-30T01:38:34.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: wait_background: for pid in $pids 2023-09-30T01:38:34.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: wait_background: wait 149302 2023-09-30T01:38:34.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/149305: /' 2023-09-30T01:38:34.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: bash -c 'sleep 2; exit 0' 2023-09-30T01:38:34.312 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: sed 's/^/149307: /' 2023-09-30T01:38:35.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 0 2023-09-30T01:38:35.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: wait_background: for pid in $pids 2023-09-30T01:38:35.314 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: wait_background: wait 149304 2023-09-30T01:38:36.315 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: run_in_background: return 0 2023-09-30T01:38:36.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: wait_background: eval 'pids='\'''\''' 2023-09-30T01:38:36.317 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: wait_background: pids= 2023-09-30T01:38:36.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: wait_background: return 0 2023-09-30T01:38:36.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: test_wait_background: '[' 0 -ne 0 ']' 2023-09-30T01:38:36.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: test_wait_background: '[' '!' -z '' ']' 2023-09-30T01:38:36.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:38:36.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_wait_for_clean td/ceph-helpers 2023-09-30T01:38:36.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: test_wait_for_clean: local dir=td/ceph-helpers 2023-09-30T01:38:36.317 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1679: test_wait_for_clean: setup td/ceph-helpers 2023-09-30T01:38:36.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:38:36.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:38:36.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:38:36.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:38:36.318 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:38:36.318 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:38:36.318 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:38:36.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:38:36.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:38:36.319 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:38:36.322 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:38:36.322 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:38:36.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:38:36.325 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:38:36.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:38:36.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:38:36.328 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:38:36.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:36.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:38:36.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:38:36.331 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:38:36.332 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:38:36.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:38:36.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:38:36.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:38:36.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:36.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:36.338 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:38:36.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:38:36.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:38:36.339 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:38:36.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:38:36.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:36.342 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:36.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:38:36.345 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:38:36.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:38:36.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:38:36.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:38:36.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1680: test_wait_for_clean: run_mon td/ceph-helpers a --osd_pool_default_size=2 2023-09-30T01:38:36.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:38:36.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:38:36.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:38:36.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:38:36.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:38:36.347 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2023-09-30T01:38:36.726 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:38:36.727 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:38:36.727 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:38:36.727 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:38:36.727 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:36.727 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:36.728 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:38:36.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2023-09-30T01:38:37.044 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:38:37.045 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:38:37.045 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:38:37.046 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:38:37.046 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:38:37.046 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:38:37.047 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:38:37.047 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:38:37.047 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:38:37.048 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:38:37.048 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:37.048 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:37.049 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:38:37.050 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:38:37.050 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:38:37.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:38:37.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:38:37.151 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:38:37.152 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:38:37.153 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:38:37.154 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:38:37.154 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:38:37.154 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:38:37.154 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:38:37.155 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:37.155 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:37.155 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:38:37.156 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:38:37.156 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:38:37.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1681: test_wait_for_clean: run_osd td/ceph-helpers 0 2023-09-30T01:38:37.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:38:37.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:38:37.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:38:37.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:38:37.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:38:37.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:38:37.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:38:37.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:38:37.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:38:37.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:38:37.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:38:37.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:38:37.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:38:37.243 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:38:37.243 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:38:37.243 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:38:37.243 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:38:37.245 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:38:37.245 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:38:37.245 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:38:37.245 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:38:37.245 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:38:37.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:38:37.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:38:37.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:38:37.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:38:37.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:38:37.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:38:37.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:38:37.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:38:37.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:38:37.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:38:37.246 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:38:37.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:38:37.247 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:38:37.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e09944b7-6115-4338-b46a-3267a3a22645 2023-09-30T01:38:37.247 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e09944b7-6115-4338-b46a-3267a3a22645' 2023-09-30T01:38:37.247 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 e09944b7-6115-4338-b46a-3267a3a22645 2023-09-30T01:38:37.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:38:37.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdfBdlBDfnDxAAowLPsfpcMR3bljCaAvt/4Q== 2023-09-30T01:38:37.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdfBdlBDfnDxAAowLPsfpcMR3bljCaAvt/4Q=="}' 2023-09-30T01:38:37.268 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e09944b7-6115-4338-b46a-3267a3a22645 -i td/ceph-helpers/0/new.json 2023-09-30T01:38:38.274 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:38:38.290 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:38:38.292 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdfBdlBDfnDxAAowLPsfpcMR3bljCaAvt/4Q== --osd-uuid e09944b7-6115-4338-b46a-3267a3a22645 2023-09-30T01:38:38.324 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:38.322+0000 7f5ad30b53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:38:38.327 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:38.326+0000 7f5ad30b53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:38:38.329 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:38.326+0000 7f5ad30b53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:38:38.330 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:38.326+0000 7f5ad30b53c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:38:38.330 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:38.326+0000 7f5ad30b53c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:38:38.330 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:38.326+0000 7f5ad30b53c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:38:38.431 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:38.426+0000 7f5ad30b53c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:38:44.157 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:38:44.157 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:38:44.159 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:38:44.159 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:38:44.160 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:38:44.403 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:38:44.403 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:38:44.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:38:44.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:38:44.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:38:44.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:38:44.434 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:44.430+0000 7f664dfc53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:38:44.439 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:44.438+0000 7f664dfc53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:38:44.441 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:44.438+0000 7f664dfc53c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:38:44.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:38:44.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:38:44.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:38:44.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:38:44.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:38:44.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:44.635 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:38:44.636 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:38:44.636 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:44.636 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:44.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:44.990 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:44.986+0000 7f664dfc53c0 -1 Falling back to public interface 2023-09-30T01:38:45.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:45.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:45.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:38:45.861 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:38:45.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:45.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:46.092 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:47.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:47.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:47.094 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:38:47.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:38:47.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:47.095 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:47.325 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:47.666 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:38:47.662+0000 7f664dfc53c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:38:48.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:48.327 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:48.327 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:38:48.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:38:48.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:48.328 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:48.684 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:49.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:49.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:49.687 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:38:49.687 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:38:49.687 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:49.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:49.914 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:50.916 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:50.916 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:50.917 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:38:50.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:38:50.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:50.917 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:51.145 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:52.146 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:52.147 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:52.147 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:38:52.147 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:38:52.147 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:52.148 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:52.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:53.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:53.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:53.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:38:53.388 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:38:53.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:53.389 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:53.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:54.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:54.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:54.619 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:38:54.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:38:54.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:54.620 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:54.852 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:55.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:55.854 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:55.854 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:38:55.855 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:38:55.855 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:55.855 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:56.082 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:57.084 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:57.084 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:57.085 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:38:57.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:38:57.085 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:57.086 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:57.380 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:58.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:58.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:58.382 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:38:58.383 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:38:58.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:58.383 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:58.611 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:38:59.612 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:38:59.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:38:59.613 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:38:59.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:38:59.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:38:59.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:38:59.839 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:00.841 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:00.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:00.842 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:39:00.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:39:00.842 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:00.843 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:39:01.070 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:02.072 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:02.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:02.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:39:02.073 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:39:02.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:02.074 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:39:02.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:03.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:03.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:03.433 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:39:03.434 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:39:03.434 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:03.434 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:39:03.660 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:04.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:04.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:04.663 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:39:04.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:39:04.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:04.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:39:04.898 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:05.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:05.900 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:05.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-09-30T01:39:05.901 INFO:tasks.workunit.client.0.smithi186.stdout:17 2023-09-30T01:39:05.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:05.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:39:06.140 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:07.142 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:07.142 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:07.142 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2023-09-30T01:39:07.142 INFO:tasks.workunit.client.0.smithi186.stdout:18 2023-09-30T01:39:07.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:07.143 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:39:07.372 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:08.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:08.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:08.375 INFO:tasks.workunit.client.0.smithi186.stdout:19 2023-09-30T01:39:08.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2023-09-30T01:39:08.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:08.375 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:39:08.610 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:09.612 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:09.612 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:09.612 INFO:tasks.workunit.client.0.smithi186.stdout:20 2023-09-30T01:39:09.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2023-09-30T01:39:09.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:09.613 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:39:09.917 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/150249,v1:127.0.0.1:6801/150249] [v2:127.0.0.1:6802/150249,v1:127.0.0.1:6803/150249] exists,up e09944b7-6115-4338-b46a-3267a3a22645 2023-09-30T01:39:09.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:39:09.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:39:09.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:39:09.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1682: test_wait_for_clean: run_mgr td/ceph-helpers x 2023-09-30T01:39:09.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:39:09.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:39:09.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:39:09.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:39:09.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:39:09.919 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:39:10.246 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:39:10.246 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:39:10.247 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:39:10.247 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:39:10.247 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:39:10.247 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:39:10.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:39:10.248 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:39:10.249 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:39:10.283 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: test_wait_for_clean: create_rbd_pool 2023-09-30T01:39:10.284 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2023-09-30T01:39:10.514 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' does not exist 2023-09-30T01:39:10.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2023-09-30T01:39:10.522 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2023-09-30T01:39:10.889 INFO:tasks.workunit.client.0.smithi186.stderr:pool 'rbd' already exists 2023-09-30T01:39:10.896 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-09-30T01:39:11.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2023-09-30T01:39:12.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: WAIT_FOR_CLEAN_TIMEOUT=1 2023-09-30T01:39:12.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: wait_for_clean 2023-09-30T01:39:12.929 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:39:12.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:39:12.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:39:12.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:39:12.930 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2023-09-30T01:39:12.930 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:39:12.931 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:39:12.931 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:39:12.931 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:39:12.931 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:39:12.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:39:12.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:39:12.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:39:12.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:39:12.986 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:39:13.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids=0 2023-09-30T01:39:13.201 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:39:13.202 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:39:13.202 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:39:13.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=0 2023-09-30T01:39:13.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 0 2023-09-30T01:39:13.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-0' 2023-09-30T01:39:13.346 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:39:13.346 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-0 2023-09-30T01:39:13.347 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:39:13.348 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:39:13.348 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-0 2023-09-30T01:39:13.349 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:39:13.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=0 2023-09-30T01:39:13.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 0' 2023-09-30T01:39:13.350 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 0 2023-09-30T01:39:13.350 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:39:13.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 0 -lt 0 2023-09-30T01:39:13.562 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:39:13.562 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:39:13.562 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:39:13.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2023-09-30T01:39:13.879 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2023-09-30T01:39:14.881 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:39:14.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:39:14.882 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:39:15.088 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2023-09-30T01:39:15.089 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2023-09-30T01:39:16.090 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:39:16.091 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:39:16.091 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:39:16.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:39:16.623 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:39:16.624 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:39:16.624 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:39:16.624 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:39:16.624 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:39:16.624 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:39:16.624 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:39:17.047 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-09-30T01:39:17.047 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:39:17.048 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:39:17.048 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:39:17.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2023-09-30T01:39:17.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2023-09-30T01:39:17.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2023-09-30T01:39:17.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2023-09-30T01:39:17.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-09-30T01:39:17.533 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2023-09-30T01:39:17.633 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-09-30T01:39:17.633 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:39:17.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:39:17.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:39:17.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:39:17.634 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:39:17.635 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:39:17.635 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:39:18.030 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-09-30T01:39:18.030 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:39:18.031 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:39:18.031 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:39:18.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2023-09-30T01:39:18.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-09-30T01:39:18.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-09-30T01:39:18.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-09-30T01:39:18.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-09-30T01:39:18.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-09-30T01:39:18.543 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-09-30T01:39:18.544 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-09-30T01:39:18.544 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-09-30T01:39:19.103 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-09-30T01:39:19.103 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-09-30T01:39:19.103 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2023-09-30T01:39:19.103 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-09-30T01:39:19.103 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2023-09-30T01:39:19.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-09-30T01:39:19.304 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:39:19.305 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:39:19.305 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:39:19.305 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:39:19.306 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:39:19.306 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:39:19.306 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:39:19.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-09-30T01:39:19.682 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:39:19.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:39:19.683 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:39:20.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2023-09-30T01:39:20.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-09-30T01:39:20.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-09-30T01:39:20.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-09-30T01:39:20.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-09-30T01:39:20.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-09-30T01:39:20.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-09-30T01:39:20.199 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-09-30T01:39:20.199 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-09-30T01:39:20.780 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-09-30T01:39:20.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-09-30T01:39:20.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2023-09-30T01:39:20.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-09-30T01:39:20.781 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2023-09-30T01:39:21.182 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-09-30T01:39:21.182 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:39:21.183 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:39:21.183 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:39:21.183 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:39:21.183 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:39:21.184 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:39:21.184 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:39:21.573 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-09-30T01:39:21.574 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:39:21.574 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:39:21.574 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:39:22.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2023-09-30T01:39:22.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-09-30T01:39:22.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-09-30T01:39:22.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-09-30T01:39:22.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-09-30T01:39:22.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-09-30T01:39:22.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-09-30T01:39:22.115 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-09-30T01:39:22.115 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-09-30T01:39:22.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-09-30T01:39:22.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-09-30T01:39:22.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2023-09-30T01:39:22.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2023-09-30T01:39:22.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2023-09-30T01:39:22.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2023-09-30T01:39:22.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:39:22.954 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:39:22.954 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:39:22.954 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:39:22.955 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:39:22.955 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:39:22.955 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:39:23.336 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2023-09-30T01:39:23.336 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:39:23.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:39:23.337 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:39:23.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2023-09-30T01:39:23.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2023-09-30T01:39:23.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2023-09-30T01:39:23.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2023-09-30T01:39:23.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2023-09-30T01:39:23.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2023-09-30T01:39:23.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2023-09-30T01:39:23.911 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2023-09-30T01:39:23.911 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2023-09-30T01:39:24.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2023-09-30T01:39:24.415 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2023-09-30T01:39:24.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2023-09-30T01:39:24.416 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2023-09-30T01:39:24.935 INFO:tasks.workunit.client.0.smithi186.stdout:{ 2023-09-30T01:39:24.935 INFO:tasks.workunit.client.0.smithi186.stdout: "cluster_fingerprint": "68ad03b2-bcd1-4a66-8f38-6cf923653977", 2023-09-30T01:39:24.935 INFO:tasks.workunit.client.0.smithi186.stdout: "version": "17.2.6-1165-g93dce5e1", 2023-09-30T01:39:24.935 INFO:tasks.workunit.client.0.smithi186.stdout: "commit": "93dce5e17918f8aca65993a7a2dc40233a47b481", 2023-09-30T01:39:24.935 INFO:tasks.workunit.client.0.smithi186.stdout: "timestamp": "2023-09-30T01:39:24.934092+0000", 2023-09-30T01:39:24.935 INFO:tasks.workunit.client.0.smithi186.stdout: "tag": "", 2023-09-30T01:39:24.935 INFO:tasks.workunit.client.0.smithi186.stdout: "health": { 2023-09-30T01:39:24.936 INFO:tasks.workunit.client.0.smithi186.stdout: "status": "HEALTH_ERR", 2023-09-30T01:39:24.936 INFO:tasks.workunit.client.0.smithi186.stdout: "checks": { 2023-09-30T01:39:24.936 INFO:tasks.workunit.client.0.smithi186.stdout: "MGR_MODULE_ERROR": { 2023-09-30T01:39:24.936 INFO:tasks.workunit.client.0.smithi186.stdout: "severity": "HEALTH_ERR", 2023-09-30T01:39:24.936 INFO:tasks.workunit.client.0.smithi186.stdout: "summary": { 2023-09-30T01:39:24.936 INFO:tasks.workunit.client.0.smithi186.stdout: "message": "Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'", 2023-09-30T01:39:24.936 INFO:tasks.workunit.client.0.smithi186.stdout: "count": 1 2023-09-30T01:39:24.936 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.936 INFO:tasks.workunit.client.0.smithi186.stdout: "detail": [ 2023-09-30T01:39:24.936 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.936 INFO:tasks.workunit.client.0.smithi186.stdout: "message": "Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'" 2023-09-30T01:39:24.937 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.937 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.937 INFO:tasks.workunit.client.0.smithi186.stdout: "muted": false 2023-09-30T01:39:24.937 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.937 INFO:tasks.workunit.client.0.smithi186.stdout: "PG_DEGRADED": { 2023-09-30T01:39:24.937 INFO:tasks.workunit.client.0.smithi186.stdout: "severity": "HEALTH_WARN", 2023-09-30T01:39:24.937 INFO:tasks.workunit.client.0.smithi186.stdout: "summary": { 2023-09-30T01:39:24.937 INFO:tasks.workunit.client.0.smithi186.stdout: "message": "Degraded data redundancy: 1/2 objects degraded (50.000%), 1 pg degraded", 2023-09-30T01:39:24.937 INFO:tasks.workunit.client.0.smithi186.stdout: "count": 1 2023-09-30T01:39:24.937 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.937 INFO:tasks.workunit.client.0.smithi186.stdout: "detail": [ 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: "muted": false 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: "TOO_FEW_OSDS": { 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: "severity": "HEALTH_WARN", 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: "summary": { 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: "count": 2 2023-09-30T01:39:24.938 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.939 INFO:tasks.workunit.client.0.smithi186.stdout: "detail": [], 2023-09-30T01:39:24.939 INFO:tasks.workunit.client.0.smithi186.stdout: "muted": false 2023-09-30T01:39:24.939 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.939 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.939 INFO:tasks.workunit.client.0.smithi186.stdout: "mutes": [] 2023-09-30T01:39:24.939 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.939 INFO:tasks.workunit.client.0.smithi186.stdout: "monmap_first_committed": 1, 2023-09-30T01:39:24.939 INFO:tasks.workunit.client.0.smithi186.stdout: "monmap_last_committed": 1, 2023-09-30T01:39:24.939 INFO:tasks.workunit.client.0.smithi186.stdout: "monmap": { 2023-09-30T01:39:24.939 INFO:tasks.workunit.client.0.smithi186.stdout: "epoch": 1, 2023-09-30T01:39:24.939 INFO:tasks.workunit.client.0.smithi186.stdout: "fsid": "779146aa-b0f9-4583-8ddd-7df6da70ad9d", 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "modified": "2023-09-30T01:38:36.375503Z", 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "created": "2023-09-30T01:38:36.375503Z", 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "min_mon_release": 17, 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "min_mon_release_name": "quincy", 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "election_strategy": 1, 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "disallowed_leaders: ": "", 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "stretch_mode": false, 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "tiebreaker_mon": "", 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "removed_ranks: ": "", 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "features": { 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "persistent": [ 2023-09-30T01:39:24.940 INFO:tasks.workunit.client.0.smithi186.stdout: "kraken", 2023-09-30T01:39:24.941 INFO:tasks.workunit.client.0.smithi186.stdout: "luminous", 2023-09-30T01:39:24.941 INFO:tasks.workunit.client.0.smithi186.stdout: "mimic", 2023-09-30T01:39:24.941 INFO:tasks.workunit.client.0.smithi186.stdout: "osdmap-prune", 2023-09-30T01:39:24.941 INFO:tasks.workunit.client.0.smithi186.stdout: "nautilus", 2023-09-30T01:39:24.941 INFO:tasks.workunit.client.0.smithi186.stdout: "octopus", 2023-09-30T01:39:24.941 INFO:tasks.workunit.client.0.smithi186.stdout: "pacific", 2023-09-30T01:39:24.941 INFO:tasks.workunit.client.0.smithi186.stdout: "elector-pinging", 2023-09-30T01:39:24.941 INFO:tasks.workunit.client.0.smithi186.stdout: "quincy" 2023-09-30T01:39:24.941 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.941 INFO:tasks.workunit.client.0.smithi186.stdout: "optional": [] 2023-09-30T01:39:24.941 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.942 INFO:tasks.workunit.client.0.smithi186.stdout: "mons": [ 2023-09-30T01:39:24.942 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.942 INFO:tasks.workunit.client.0.smithi186.stdout: "rank": 0, 2023-09-30T01:39:24.942 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "a", 2023-09-30T01:39:24.942 INFO:tasks.workunit.client.0.smithi186.stdout: "public_addrs": { 2023-09-30T01:39:24.942 INFO:tasks.workunit.client.0.smithi186.stdout: "addrvec": [ 2023-09-30T01:39:24.942 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.942 INFO:tasks.workunit.client.0.smithi186.stdout: "type": "v2", 2023-09-30T01:39:24.942 INFO:tasks.workunit.client.0.smithi186.stdout: "addr": "127.0.0.1:7109", 2023-09-30T01:39:24.942 INFO:tasks.workunit.client.0.smithi186.stdout: "nonce": 0 2023-09-30T01:39:24.942 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: "addr": "127.0.0.1:7109/0", 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: "public_addr": "127.0.0.1:7109/0", 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: "priority": 0, 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: "weight": 0, 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: "crush_location": "{}" 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: "quorum": [ 2023-09-30T01:39:24.943 INFO:tasks.workunit.client.0.smithi186.stdout: 0 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: "osdmap": { 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: "epoch": 12, 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: "fsid": "779146aa-b0f9-4583-8ddd-7df6da70ad9d", 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: "created": "2023-09-30T01:38:37.324750+0000", 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: "modified": "2023-09-30T01:39:12.631939+0000", 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: "last_up_change": "2023-09-30T01:39:09.385260+0000", 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: "last_in_change": "2023-09-30T01:38:38.065960+0000", 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: "flags_num": 5799936, 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: "flags_set": [ 2023-09-30T01:39:24.944 INFO:tasks.workunit.client.0.smithi186.stdout: "pglog_hardlimit", 2023-09-30T01:39:24.945 INFO:tasks.workunit.client.0.smithi186.stdout: "purged_snapdirs", 2023-09-30T01:39:24.945 INFO:tasks.workunit.client.0.smithi186.stdout: "recovery_deletes", 2023-09-30T01:39:24.945 INFO:tasks.workunit.client.0.smithi186.stdout: "sortbitwise" 2023-09-30T01:39:24.945 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.945 INFO:tasks.workunit.client.0.smithi186.stdout: "crush_version": 3, 2023-09-30T01:39:24.945 INFO:tasks.workunit.client.0.smithi186.stdout: "full_ratio": 0.99000000953674316, 2023-09-30T01:39:24.945 INFO:tasks.workunit.client.0.smithi186.stdout: "backfillfull_ratio": 0.99000000953674316, 2023-09-30T01:39:24.945 INFO:tasks.workunit.client.0.smithi186.stdout: "nearfull_ratio": 0.85000002384185791, 2023-09-30T01:39:24.945 INFO:tasks.workunit.client.0.smithi186.stdout: "cluster_snapshot": "", 2023-09-30T01:39:24.945 INFO:tasks.workunit.client.0.smithi186.stdout: "pool_max": 1, 2023-09-30T01:39:24.946 INFO:tasks.workunit.client.0.smithi186.stdout: "max_osd": 1, 2023-09-30T01:39:24.946 INFO:tasks.workunit.client.0.smithi186.stdout: "require_min_compat_client": "luminous", 2023-09-30T01:39:24.946 INFO:tasks.workunit.client.0.smithi186.stdout: "min_compat_client": "jewel", 2023-09-30T01:39:24.946 INFO:tasks.workunit.client.0.smithi186.stdout: "require_osd_release": "quincy", 2023-09-30T01:39:24.946 INFO:tasks.workunit.client.0.smithi186.stdout: "pools": [ 2023-09-30T01:39:24.946 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.946 INFO:tasks.workunit.client.0.smithi186.stdout: "pool": 1, 2023-09-30T01:39:24.946 INFO:tasks.workunit.client.0.smithi186.stdout: "pool_name": "rbd", 2023-09-30T01:39:24.946 INFO:tasks.workunit.client.0.smithi186.stdout: "create_time": "2023-09-30T01:39:10.708586+0000", 2023-09-30T01:39:24.946 INFO:tasks.workunit.client.0.smithi186.stdout: "flags": 8193, 2023-09-30T01:39:24.947 INFO:tasks.workunit.client.0.smithi186.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2023-09-30T01:39:24.947 INFO:tasks.workunit.client.0.smithi186.stdout: "type": 1, 2023-09-30T01:39:24.947 INFO:tasks.workunit.client.0.smithi186.stdout: "size": 2, 2023-09-30T01:39:24.947 INFO:tasks.workunit.client.0.smithi186.stdout: "min_size": 1, 2023-09-30T01:39:24.947 INFO:tasks.workunit.client.0.smithi186.stdout: "crush_rule": 0, 2023-09-30T01:39:24.947 INFO:tasks.workunit.client.0.smithi186.stdout: "peering_crush_bucket_count": 0, 2023-09-30T01:39:24.947 INFO:tasks.workunit.client.0.smithi186.stdout: "peering_crush_bucket_target": 0, 2023-09-30T01:39:24.947 INFO:tasks.workunit.client.0.smithi186.stdout: "peering_crush_bucket_barrier": 0, 2023-09-30T01:39:24.947 INFO:tasks.workunit.client.0.smithi186.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-09-30T01:39:24.947 INFO:tasks.workunit.client.0.smithi186.stdout: "object_hash": 2, 2023-09-30T01:39:24.948 INFO:tasks.workunit.client.0.smithi186.stdout: "pg_autoscale_mode": "off", 2023-09-30T01:39:24.948 INFO:tasks.workunit.client.0.smithi186.stdout: "pg_num": 4, 2023-09-30T01:39:24.948 INFO:tasks.workunit.client.0.smithi186.stdout: "pg_placement_num": 4, 2023-09-30T01:39:24.948 INFO:tasks.workunit.client.0.smithi186.stdout: "pg_placement_num_target": 4, 2023-09-30T01:39:24.948 INFO:tasks.workunit.client.0.smithi186.stdout: "pg_num_target": 4, 2023-09-30T01:39:24.948 INFO:tasks.workunit.client.0.smithi186.stdout: "pg_num_pending": 4, 2023-09-30T01:39:24.948 INFO:tasks.workunit.client.0.smithi186.stdout: "last_pg_merge_meta": { 2023-09-30T01:39:24.948 INFO:tasks.workunit.client.0.smithi186.stdout: "source_pgid": "0.0", 2023-09-30T01:39:24.948 INFO:tasks.workunit.client.0.smithi186.stdout: "ready_epoch": 0, 2023-09-30T01:39:24.948 INFO:tasks.workunit.client.0.smithi186.stdout: "last_epoch_started": 0, 2023-09-30T01:39:24.949 INFO:tasks.workunit.client.0.smithi186.stdout: "last_epoch_clean": 0, 2023-09-30T01:39:24.949 INFO:tasks.workunit.client.0.smithi186.stdout: "source_version": "0'0", 2023-09-30T01:39:24.949 INFO:tasks.workunit.client.0.smithi186.stdout: "target_version": "0'0" 2023-09-30T01:39:24.949 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.949 INFO:tasks.workunit.client.0.smithi186.stdout: "last_change": "12", 2023-09-30T01:39:24.949 INFO:tasks.workunit.client.0.smithi186.stdout: "last_force_op_resend": "0", 2023-09-30T01:39:24.949 INFO:tasks.workunit.client.0.smithi186.stdout: "last_force_op_resend_prenautilus": "0", 2023-09-30T01:39:24.949 INFO:tasks.workunit.client.0.smithi186.stdout: "last_force_op_resend_preluminous": "0", 2023-09-30T01:39:24.949 INFO:tasks.workunit.client.0.smithi186.stdout: "auid": 0, 2023-09-30T01:39:24.949 INFO:tasks.workunit.client.0.smithi186.stdout: "snap_mode": "selfmanaged", 2023-09-30T01:39:24.950 INFO:tasks.workunit.client.0.smithi186.stdout: "snap_seq": 3, 2023-09-30T01:39:24.950 INFO:tasks.workunit.client.0.smithi186.stdout: "snap_epoch": 12, 2023-09-30T01:39:24.950 INFO:tasks.workunit.client.0.smithi186.stdout: "pool_snaps": [], 2023-09-30T01:39:24.950 INFO:tasks.workunit.client.0.smithi186.stdout: "removed_snaps": "[]", 2023-09-30T01:39:24.950 INFO:tasks.workunit.client.0.smithi186.stdout: "quota_max_bytes": 0, 2023-09-30T01:39:24.950 INFO:tasks.workunit.client.0.smithi186.stdout: "quota_max_objects": 0, 2023-09-30T01:39:24.950 INFO:tasks.workunit.client.0.smithi186.stdout: "tiers": [], 2023-09-30T01:39:24.950 INFO:tasks.workunit.client.0.smithi186.stdout: "tier_of": -1, 2023-09-30T01:39:24.950 INFO:tasks.workunit.client.0.smithi186.stdout: "read_tier": -1, 2023-09-30T01:39:24.950 INFO:tasks.workunit.client.0.smithi186.stdout: "write_tier": -1, 2023-09-30T01:39:24.951 INFO:tasks.workunit.client.0.smithi186.stdout: "cache_mode": "none", 2023-09-30T01:39:24.951 INFO:tasks.workunit.client.0.smithi186.stdout: "target_max_bytes": 0, 2023-09-30T01:39:24.951 INFO:tasks.workunit.client.0.smithi186.stdout: "target_max_objects": 0, 2023-09-30T01:39:24.951 INFO:tasks.workunit.client.0.smithi186.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-09-30T01:39:24.951 INFO:tasks.workunit.client.0.smithi186.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-09-30T01:39:24.951 INFO:tasks.workunit.client.0.smithi186.stdout: "cache_target_full_ratio_micro": 800000, 2023-09-30T01:39:24.951 INFO:tasks.workunit.client.0.smithi186.stdout: "cache_min_flush_age": 0, 2023-09-30T01:39:24.951 INFO:tasks.workunit.client.0.smithi186.stdout: "cache_min_evict_age": 0, 2023-09-30T01:39:24.951 INFO:tasks.workunit.client.0.smithi186.stdout: "erasure_code_profile": "", 2023-09-30T01:39:24.951 INFO:tasks.workunit.client.0.smithi186.stdout: "hit_set_params": { 2023-09-30T01:39:24.952 INFO:tasks.workunit.client.0.smithi186.stdout: "type": "none" 2023-09-30T01:39:24.952 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.952 INFO:tasks.workunit.client.0.smithi186.stdout: "hit_set_period": 0, 2023-09-30T01:39:24.952 INFO:tasks.workunit.client.0.smithi186.stdout: "hit_set_count": 0, 2023-09-30T01:39:24.952 INFO:tasks.workunit.client.0.smithi186.stdout: "use_gmt_hitset": true, 2023-09-30T01:39:24.952 INFO:tasks.workunit.client.0.smithi186.stdout: "min_read_recency_for_promote": 0, 2023-09-30T01:39:24.952 INFO:tasks.workunit.client.0.smithi186.stdout: "min_write_recency_for_promote": 0, 2023-09-30T01:39:24.952 INFO:tasks.workunit.client.0.smithi186.stdout: "hit_set_grade_decay_rate": 0, 2023-09-30T01:39:24.952 INFO:tasks.workunit.client.0.smithi186.stdout: "hit_set_search_last_n": 0, 2023-09-30T01:39:24.952 INFO:tasks.workunit.client.0.smithi186.stdout: "grade_table": [], 2023-09-30T01:39:24.953 INFO:tasks.workunit.client.0.smithi186.stdout: "stripe_width": 0, 2023-09-30T01:39:24.953 INFO:tasks.workunit.client.0.smithi186.stdout: "expected_num_objects": 0, 2023-09-30T01:39:24.953 INFO:tasks.workunit.client.0.smithi186.stdout: "fast_read": false, 2023-09-30T01:39:24.953 INFO:tasks.workunit.client.0.smithi186.stdout: "options": {}, 2023-09-30T01:39:24.953 INFO:tasks.workunit.client.0.smithi186.stdout: "application_metadata": { 2023-09-30T01:39:24.953 INFO:tasks.workunit.client.0.smithi186.stdout: "rbd": {} 2023-09-30T01:39:24.953 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.953 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.953 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.953 INFO:tasks.workunit.client.0.smithi186.stdout: "osds": [ 2023-09-30T01:39:24.954 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.954 INFO:tasks.workunit.client.0.smithi186.stdout: "osd": 0, 2023-09-30T01:39:24.954 INFO:tasks.workunit.client.0.smithi186.stdout: "uuid": "e09944b7-6115-4338-b46a-3267a3a22645", 2023-09-30T01:39:24.954 INFO:tasks.workunit.client.0.smithi186.stdout: "up": 1, 2023-09-30T01:39:24.954 INFO:tasks.workunit.client.0.smithi186.stdout: "in": 1, 2023-09-30T01:39:24.954 INFO:tasks.workunit.client.0.smithi186.stdout: "weight": 1, 2023-09-30T01:39:24.954 INFO:tasks.workunit.client.0.smithi186.stdout: "primary_affinity": 1, 2023-09-30T01:39:24.954 INFO:tasks.workunit.client.0.smithi186.stdout: "last_clean_begin": 0, 2023-09-30T01:39:24.954 INFO:tasks.workunit.client.0.smithi186.stdout: "last_clean_end": 0, 2023-09-30T01:39:24.954 INFO:tasks.workunit.client.0.smithi186.stdout: "up_from": 5, 2023-09-30T01:39:24.955 INFO:tasks.workunit.client.0.smithi186.stdout: "up_thru": 7, 2023-09-30T01:39:24.955 INFO:tasks.workunit.client.0.smithi186.stdout: "down_at": 0, 2023-09-30T01:39:24.955 INFO:tasks.workunit.client.0.smithi186.stdout: "lost_at": 0, 2023-09-30T01:39:24.955 INFO:tasks.workunit.client.0.smithi186.stdout: "public_addrs": { 2023-09-30T01:39:24.955 INFO:tasks.workunit.client.0.smithi186.stdout: "addrvec": [ 2023-09-30T01:39:24.955 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.955 INFO:tasks.workunit.client.0.smithi186.stdout: "type": "v2", 2023-09-30T01:39:24.955 INFO:tasks.workunit.client.0.smithi186.stdout: "addr": "127.0.0.1:6800", 2023-09-30T01:39:24.955 INFO:tasks.workunit.client.0.smithi186.stdout: "nonce": 150249 2023-09-30T01:39:24.955 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.956 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.956 INFO:tasks.workunit.client.0.smithi186.stdout: "type": "v1", 2023-09-30T01:39:24.956 INFO:tasks.workunit.client.0.smithi186.stdout: "addr": "127.0.0.1:6801", 2023-09-30T01:39:24.956 INFO:tasks.workunit.client.0.smithi186.stdout: "nonce": 150249 2023-09-30T01:39:24.956 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.956 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.956 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.956 INFO:tasks.workunit.client.0.smithi186.stdout: "cluster_addrs": { 2023-09-30T01:39:24.956 INFO:tasks.workunit.client.0.smithi186.stdout: "addrvec": [ 2023-09-30T01:39:24.956 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.957 INFO:tasks.workunit.client.0.smithi186.stdout: "type": "v2", 2023-09-30T01:39:24.957 INFO:tasks.workunit.client.0.smithi186.stdout: "addr": "127.0.0.1:6802", 2023-09-30T01:39:24.957 INFO:tasks.workunit.client.0.smithi186.stdout: "nonce": 150249 2023-09-30T01:39:24.957 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.957 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.957 INFO:tasks.workunit.client.0.smithi186.stdout: "type": "v1", 2023-09-30T01:39:24.957 INFO:tasks.workunit.client.0.smithi186.stdout: "addr": "127.0.0.1:6803", 2023-09-30T01:39:24.957 INFO:tasks.workunit.client.0.smithi186.stdout: "nonce": 150249 2023-09-30T01:39:24.957 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.957 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.957 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: "heartbeat_back_addrs": { 2023-09-30T01:39:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: "addrvec": [ 2023-09-30T01:39:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: "type": "v2", 2023-09-30T01:39:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: "addr": "127.0.0.1:6806", 2023-09-30T01:39:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: "nonce": 150249 2023-09-30T01:39:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: "type": "v1", 2023-09-30T01:39:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: "addr": "127.0.0.1:6807", 2023-09-30T01:39:24.958 INFO:tasks.workunit.client.0.smithi186.stdout: "nonce": 150249 2023-09-30T01:39:24.959 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.959 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.959 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.959 INFO:tasks.workunit.client.0.smithi186.stdout: "heartbeat_front_addrs": { 2023-09-30T01:39:24.959 INFO:tasks.workunit.client.0.smithi186.stdout: "addrvec": [ 2023-09-30T01:39:24.959 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.959 INFO:tasks.workunit.client.0.smithi186.stdout: "type": "v2", 2023-09-30T01:39:24.959 INFO:tasks.workunit.client.0.smithi186.stdout: "addr": "127.0.0.1:6804", 2023-09-30T01:39:24.959 INFO:tasks.workunit.client.0.smithi186.stdout: "nonce": 150249 2023-09-30T01:39:24.959 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.959 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: "type": "v1", 2023-09-30T01:39:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: "addr": "127.0.0.1:6805", 2023-09-30T01:39:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: "nonce": 150249 2023-09-30T01:39:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: "public_addr": "127.0.0.1:6801/150249", 2023-09-30T01:39:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: "cluster_addr": "127.0.0.1:6803/150249", 2023-09-30T01:39:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: "heartbeat_back_addr": "127.0.0.1:6807/150249", 2023-09-30T01:39:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: "heartbeat_front_addr": "127.0.0.1:6805/150249", 2023-09-30T01:39:24.960 INFO:tasks.workunit.client.0.smithi186.stdout: "state": [ 2023-09-30T01:39:24.961 INFO:tasks.workunit.client.0.smithi186.stdout: "exists", 2023-09-30T01:39:24.961 INFO:tasks.workunit.client.0.smithi186.stdout: "up" 2023-09-30T01:39:24.961 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.961 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.961 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.961 INFO:tasks.workunit.client.0.smithi186.stdout: "osd_xinfo": [ 2023-09-30T01:39:24.961 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.961 INFO:tasks.workunit.client.0.smithi186.stdout: "osd": 0, 2023-09-30T01:39:24.961 INFO:tasks.workunit.client.0.smithi186.stdout: "down_stamp": "0.000000", 2023-09-30T01:39:24.961 INFO:tasks.workunit.client.0.smithi186.stdout: "laggy_probability": 0, 2023-09-30T01:39:24.961 INFO:tasks.workunit.client.0.smithi186.stdout: "laggy_interval": 0, 2023-09-30T01:39:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: "features": 4540138320759226367, 2023-09-30T01:39:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: "old_weight": 0, 2023-09-30T01:39:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: "last_purged_snaps_scrub": "2023-09-30T01:39:09.043235+0000", 2023-09-30T01:39:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: "dead_epoch": 0 2023-09-30T01:39:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: "pg_upmap": [], 2023-09-30T01:39:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: "pg_upmap_items": [], 2023-09-30T01:39:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: "pg_temp": [], 2023-09-30T01:39:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: "primary_temp": [], 2023-09-30T01:39:24.962 INFO:tasks.workunit.client.0.smithi186.stdout: "blocklist": {}, 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: "range_blocklist": {}, 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: "erasure_code_profiles": { 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: "default": { 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: "k": "2", 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: "m": "2", 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: "plugin": "jerasure", 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: "technique": "reed_sol_van" 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: "removed_snaps_queue": [ 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.963 INFO:tasks.workunit.client.0.smithi186.stdout: "pool": 1, 2023-09-30T01:39:24.964 INFO:tasks.workunit.client.0.smithi186.stdout: "snaps": [ 2023-09-30T01:39:24.964 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.964 INFO:tasks.workunit.client.0.smithi186.stdout: "begin": 2, 2023-09-30T01:39:24.964 INFO:tasks.workunit.client.0.smithi186.stdout: "length": 1 2023-09-30T01:39:24.964 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.964 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.964 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.964 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.964 INFO:tasks.workunit.client.0.smithi186.stdout: "new_removed_snaps": [ 2023-09-30T01:39:24.964 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.964 INFO:tasks.workunit.client.0.smithi186.stdout: "pool": 1, 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: "snaps": [ 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: "begin": 2, 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: "length": 1 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: "new_purged_snaps": [], 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: "crush_node_flags": {}, 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: "device_class_flags": {}, 2023-09-30T01:39:24.965 INFO:tasks.workunit.client.0.smithi186.stdout: "stretch_mode": { 2023-09-30T01:39:24.966 INFO:tasks.workunit.client.0.smithi186.stdout: "stretch_mode_enabled": false, 2023-09-30T01:39:24.966 INFO:tasks.workunit.client.0.smithi186.stdout: "stretch_bucket_count": 0, 2023-09-30T01:39:24.966 INFO:tasks.workunit.client.0.smithi186.stdout: "degraded_stretch_mode": 0, 2023-09-30T01:39:24.966 INFO:tasks.workunit.client.0.smithi186.stdout: "recovering_stretch_mode": 0, 2023-09-30T01:39:24.966 INFO:tasks.workunit.client.0.smithi186.stdout: "stretch_mode_bucket": 0 2023-09-30T01:39:24.966 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.966 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.966 INFO:tasks.workunit.client.0.smithi186.stdout: "osd_metadata": [ 2023-09-30T01:39:24.966 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.966 INFO:tasks.workunit.client.0.smithi186.stdout: "id": 0, 2023-09-30T01:39:24.966 INFO:tasks.workunit.client.0.smithi186.stdout: "arch": "x86_64", 2023-09-30T01:39:24.967 INFO:tasks.workunit.client.0.smithi186.stdout: "back_addr": "[v2:127.0.0.1:6802/150249,v1:127.0.0.1:6803/150249]", 2023-09-30T01:39:24.967 INFO:tasks.workunit.client.0.smithi186.stdout: "back_iface": "", 2023-09-30T01:39:24.967 INFO:tasks.workunit.client.0.smithi186.stdout: "bluefs": "1", 2023-09-30T01:39:24.967 INFO:tasks.workunit.client.0.smithi186.stdout: "bluefs_dedicated_db": "0", 2023-09-30T01:39:24.967 INFO:tasks.workunit.client.0.smithi186.stdout: "bluefs_dedicated_wal": "0", 2023-09-30T01:39:24.967 INFO:tasks.workunit.client.0.smithi186.stdout: "bluefs_single_shared_device": "1", 2023-09-30T01:39:24.967 INFO:tasks.workunit.client.0.smithi186.stdout: "bluestore_bdev_access_mode": "file", 2023-09-30T01:39:24.967 INFO:tasks.workunit.client.0.smithi186.stdout: "bluestore_bdev_block_size": "4096", 2023-09-30T01:39:24.967 INFO:tasks.workunit.client.0.smithi186.stdout: "bluestore_bdev_devices": "sda", 2023-09-30T01:39:24.967 INFO:tasks.workunit.client.0.smithi186.stdout: "bluestore_bdev_driver": "KernelDevice", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "bluestore_bdev_optimal_io_size": "0", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "bluestore_bdev_rotational": "1", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "bluestore_bdev_size": "107374182400", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "bluestore_bdev_support_discard": "0", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "bluestore_bdev_type": "hdd", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "bluestore_min_alloc_size": "4096", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "ceph_release": "quincy", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "ceph_version": "ceph version 17.2.6-1165-g93dce5e1 (93dce5e17918f8aca65993a7a2dc40233a47b481) quincy (stable)", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "ceph_version_short": "17.2.6-1165-g93dce5e1", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "ceph_version_when_created": "ceph version 17.2.6-1165-g93dce5e1 (93dce5e17918f8aca65993a7a2dc40233a47b481) quincy (stable)", 2023-09-30T01:39:24.968 INFO:tasks.workunit.client.0.smithi186.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-09-30T01:39:24.969 INFO:tasks.workunit.client.0.smithi186.stdout: "created_at": "2023-09-30T01:38:43.477913Z", 2023-09-30T01:39:24.969 INFO:tasks.workunit.client.0.smithi186.stdout: "default_device_class": "hdd", 2023-09-30T01:39:24.969 INFO:tasks.workunit.client.0.smithi186.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XTX2", 2023-09-30T01:39:24.969 INFO:tasks.workunit.client.0.smithi186.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2023-09-30T01:39:24.969 INFO:tasks.workunit.client.0.smithi186.stdout: "devices": "sda", 2023-09-30T01:39:24.969 INFO:tasks.workunit.client.0.smithi186.stdout: "distro": "ubuntu", 2023-09-30T01:39:24.969 INFO:tasks.workunit.client.0.smithi186.stdout: "distro_description": "Ubuntu 20.04.4 LTS", 2023-09-30T01:39:24.969 INFO:tasks.workunit.client.0.smithi186.stdout: "distro_version": "20.04", 2023-09-30T01:39:24.969 INFO:tasks.workunit.client.0.smithi186.stdout: "front_addr": "[v2:127.0.0.1:6800/150249,v1:127.0.0.1:6801/150249]", 2023-09-30T01:39:24.969 INFO:tasks.workunit.client.0.smithi186.stdout: "front_iface": "", 2023-09-30T01:39:24.969 INFO:tasks.workunit.client.0.smithi186.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/150249,v1:127.0.0.1:6807/150249]", 2023-09-30T01:39:24.970 INFO:tasks.workunit.client.0.smithi186.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/150249,v1:127.0.0.1:6805/150249]", 2023-09-30T01:39:24.970 INFO:tasks.workunit.client.0.smithi186.stdout: "hostname": "smithi186", 2023-09-30T01:39:24.970 INFO:tasks.workunit.client.0.smithi186.stdout: "journal_rotational": "1", 2023-09-30T01:39:24.970 INFO:tasks.workunit.client.0.smithi186.stdout: "kernel_description": "#180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023", 2023-09-30T01:39:24.970 INFO:tasks.workunit.client.0.smithi186.stdout: "kernel_version": "5.4.0-163-generic", 2023-09-30T01:39:24.970 INFO:tasks.workunit.client.0.smithi186.stdout: "mem_swap_kb": "2097148", 2023-09-30T01:39:24.970 INFO:tasks.workunit.client.0.smithi186.stdout: "mem_total_kb": "32755048", 2023-09-30T01:39:24.970 INFO:tasks.workunit.client.0.smithi186.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2023-09-30T01:39:24.970 INFO:tasks.workunit.client.0.smithi186.stdout: "objectstore_numa_unknown_devices": "sda", 2023-09-30T01:39:24.970 INFO:tasks.workunit.client.0.smithi186.stdout: "os": "Linux", 2023-09-30T01:39:24.970 INFO:tasks.workunit.client.0.smithi186.stdout: "osd_data": "td/ceph-helpers/0", 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: "osd_objectstore": "bluestore", 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: "osdspec_affinity": "", 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: "rotational": "1" 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: "osdmap_clean_epochs": { 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: "min_last_epoch_clean": 0, 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: "last_epoch_clean": { 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: "per_pool": [] 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: "osd_epochs": [ 2023-09-30T01:39:24.971 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.972 INFO:tasks.workunit.client.0.smithi186.stdout: "id": 0, 2023-09-30T01:39:24.972 INFO:tasks.workunit.client.0.smithi186.stdout: "epoch": 5 2023-09-30T01:39:24.972 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.972 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.972 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.972 INFO:tasks.workunit.client.0.smithi186.stdout: "osdmap_first_committed": 1, 2023-09-30T01:39:24.972 INFO:tasks.workunit.client.0.smithi186.stdout: "osdmap_last_committed": 12, 2023-09-30T01:39:24.972 INFO:tasks.workunit.client.0.smithi186.stdout: "crushmap": { 2023-09-30T01:39:24.972 INFO:tasks.workunit.client.0.smithi186.stdout: "devices": [ 2023-09-30T01:39:24.972 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.972 INFO:tasks.workunit.client.0.smithi186.stdout: "id": 0, 2023-09-30T01:39:24.973 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "osd.0", 2023-09-30T01:39:24.973 INFO:tasks.workunit.client.0.smithi186.stdout: "class": "hdd" 2023-09-30T01:39:24.973 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.973 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.973 INFO:tasks.workunit.client.0.smithi186.stdout: "types": [ 2023-09-30T01:39:24.973 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.973 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 0, 2023-09-30T01:39:24.973 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "osd" 2023-09-30T01:39:24.973 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.973 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.973 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 1, 2023-09-30T01:39:24.974 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "host" 2023-09-30T01:39:24.974 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.974 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.974 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 2, 2023-09-30T01:39:24.974 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "chassis" 2023-09-30T01:39:24.974 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.974 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.974 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 3, 2023-09-30T01:39:24.974 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "rack" 2023-09-30T01:39:24.974 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.975 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.975 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 4, 2023-09-30T01:39:24.975 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "row" 2023-09-30T01:39:24.975 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.975 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.975 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 5, 2023-09-30T01:39:24.975 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "pdu" 2023-09-30T01:39:24.975 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.975 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.975 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 6, 2023-09-30T01:39:24.975 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "pod" 2023-09-30T01:39:24.976 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.976 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.976 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 7, 2023-09-30T01:39:24.976 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "room" 2023-09-30T01:39:24.976 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.976 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.976 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 8, 2023-09-30T01:39:24.976 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "datacenter" 2023-09-30T01:39:24.976 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.976 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.976 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 9, 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "zone" 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 10, 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "region" 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 11, 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "root" 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.977 INFO:tasks.workunit.client.0.smithi186.stdout: "buckets": [ 2023-09-30T01:39:24.978 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.978 INFO:tasks.workunit.client.0.smithi186.stdout: "id": -1, 2023-09-30T01:39:24.978 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "default", 2023-09-30T01:39:24.978 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 11, 2023-09-30T01:39:24.978 INFO:tasks.workunit.client.0.smithi186.stdout: "type_name": "root", 2023-09-30T01:39:24.978 INFO:tasks.workunit.client.0.smithi186.stdout: "weight": 6402, 2023-09-30T01:39:24.978 INFO:tasks.workunit.client.0.smithi186.stdout: "alg": "straw2", 2023-09-30T01:39:24.978 INFO:tasks.workunit.client.0.smithi186.stdout: "hash": "rjenkins1", 2023-09-30T01:39:24.978 INFO:tasks.workunit.client.0.smithi186.stdout: "items": [ 2023-09-30T01:39:24.978 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.978 INFO:tasks.workunit.client.0.smithi186.stdout: "id": -3, 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: "weight": 6402, 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: "pos": 0 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: "id": -2, 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "default~hdd", 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 11, 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: "type_name": "root", 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: "weight": 6402, 2023-09-30T01:39:24.979 INFO:tasks.workunit.client.0.smithi186.stdout: "alg": "straw2", 2023-09-30T01:39:24.980 INFO:tasks.workunit.client.0.smithi186.stdout: "hash": "rjenkins1", 2023-09-30T01:39:24.980 INFO:tasks.workunit.client.0.smithi186.stdout: "items": [ 2023-09-30T01:39:24.980 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.980 INFO:tasks.workunit.client.0.smithi186.stdout: "id": -4, 2023-09-30T01:39:24.980 INFO:tasks.workunit.client.0.smithi186.stdout: "weight": 6402, 2023-09-30T01:39:24.980 INFO:tasks.workunit.client.0.smithi186.stdout: "pos": 0 2023-09-30T01:39:24.980 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.980 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.980 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.980 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.980 INFO:tasks.workunit.client.0.smithi186.stdout: "id": -3, 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "smithi186", 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 1, 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: "type_name": "host", 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: "weight": 6402, 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: "alg": "straw2", 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: "hash": "rjenkins1", 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: "items": [ 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: "id": 0, 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: "weight": 6402, 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: "pos": 0 2023-09-30T01:39:24.981 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.982 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.982 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.982 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.982 INFO:tasks.workunit.client.0.smithi186.stdout: "id": -4, 2023-09-30T01:39:24.982 INFO:tasks.workunit.client.0.smithi186.stdout: "name": "smithi186~hdd", 2023-09-30T01:39:24.982 INFO:tasks.workunit.client.0.smithi186.stdout: "type_id": 1, 2023-09-30T01:39:24.982 INFO:tasks.workunit.client.0.smithi186.stdout: "type_name": "host", 2023-09-30T01:39:24.982 INFO:tasks.workunit.client.0.smithi186.stdout: "weight": 6402, 2023-09-30T01:39:24.982 INFO:tasks.workunit.client.0.smithi186.stdout: "alg": "straw2", 2023-09-30T01:39:24.982 INFO:tasks.workunit.client.0.smithi186.stdout: "hash": "rjenkins1", 2023-09-30T01:39:24.982 INFO:tasks.workunit.client.0.smithi186.stdout: "items": [ 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: "id": 0, 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: "weight": 6402, 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: "pos": 0 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: "rules": [ 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: "rule_id": 0, 2023-09-30T01:39:24.983 INFO:tasks.workunit.client.0.smithi186.stdout: "rule_name": "replicated_rule", 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: "type": 1, 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: "steps": [ 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: "op": "take", 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: "item": -1, 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: "item_name": "default" 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: "op": "choose_firstn", 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: "num": 0, 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: "type": "osd" 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.984 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: "op": "emit" 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: ] 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: "tunables": { 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: "choose_local_tries": 0, 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: "choose_local_fallback_tries": 0, 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: "choose_total_tries": 50, 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: "chooseleaf_descend_once": 1, 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: "chooseleaf_vary_r": 1, 2023-09-30T01:39:24.985 INFO:tasks.workunit.client.0.smithi186.stdout: "chooseleaf_stable": 1, 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "straw_calc_version": 1, 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "allowed_bucket_algs": 54, 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "profile": "jewel", 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "optimal_tunables": 1, 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "legacy_tunables": 0, 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "minimum_required_version": "jewel", 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "require_feature_tunables": 1, 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "require_feature_tunables2": 1, 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "has_v2_rules": 0, 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "require_feature_tunables3": 1, 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "has_v3_rules": 0, 2023-09-30T01:39:24.986 INFO:tasks.workunit.client.0.smithi186.stdout: "has_v4_buckets": 1, 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: "require_feature_tunables5": 1, 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: "has_v5_rules": 0 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: "choose_args": {} 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: "fsmap": { 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: "epoch": 1, 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: "default_fscid": -1, 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: "compat": { 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: "compat": {}, 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: "ro_compat": {}, 2023-09-30T01:39:24.987 INFO:tasks.workunit.client.0.smithi186.stdout: "incompat": { 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: "feature_1": "base v0.20", 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: "feature_2": "client writeable ranges", 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: "feature_3": "default file layouts on dirs", 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: "feature_4": "dir inode in separate object", 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: "feature_5": "mds uses versioned encoding", 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: "feature_6": "dirfrag is stored in omap", 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: "feature_8": "no anchor table", 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: "feature_9": "file layout v2", 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: "feature_10": "snaprealm v2" 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.988 INFO:tasks.workunit.client.0.smithi186.stdout: "feature_flags": { 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: "enable_multiple": true, 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: "ever_enabled_multiple": true 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: "standbys": [], 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: "filesystems": [] 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: "mdsmap_first_committed": 1, 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: "mdsmap_last_committed": 1, 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: "auth": { 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: "first_committed": 1, 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: "last_committed": 3, 2023-09-30T01:39:24.989 INFO:tasks.workunit.client.0.smithi186.stdout: "num_secrets": 1 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: "num_pg": 4, 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: "num_pg_active": 4, 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: "num_pg_unknown": 0, 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: "num_osd": 1, 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: "pool_sum": { 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: "stat_sum": { 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: "num_bytes": 19, 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects": 1, 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: "num_object_clones": 0, 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: "num_object_copies": 2, 2023-09-30T01:39:24.990 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_missing_on_primary": 0, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_missing": 0, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_degraded": 1, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_misplaced": 0, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_unfound": 0, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_dirty": 1, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_whiteouts": 0, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_read": 0, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_read_kb": 0, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_write": 2, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_write_kb": 2, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_scrub_errors": 0, 2023-09-30T01:39:24.991 INFO:tasks.workunit.client.0.smithi186.stdout: "num_shallow_scrub_errors": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_deep_scrub_errors": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_recovered": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_bytes_recovered": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_keys_recovered": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_omap": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_hit_set_archive": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_bytes_hit_set_archive": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_flush": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_flush_kb": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_evict": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_evict_kb": 0, 2023-09-30T01:39:24.992 INFO:tasks.workunit.client.0.smithi186.stdout: "num_promote": 0, 2023-09-30T01:39:24.993 INFO:tasks.workunit.client.0.smithi186.stdout: "num_flush_mode_high": 0, 2023-09-30T01:39:24.993 INFO:tasks.workunit.client.0.smithi186.stdout: "num_flush_mode_low": 0, 2023-09-30T01:39:24.993 INFO:tasks.workunit.client.0.smithi186.stdout: "num_evict_mode_some": 0, 2023-09-30T01:39:24.993 INFO:tasks.workunit.client.0.smithi186.stdout: "num_evict_mode_full": 0, 2023-09-30T01:39:24.993 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_pinned": 0, 2023-09-30T01:39:24.993 INFO:tasks.workunit.client.0.smithi186.stdout: "num_legacy_snapsets": 0, 2023-09-30T01:39:24.993 INFO:tasks.workunit.client.0.smithi186.stdout: "num_large_omap_objects": 0, 2023-09-30T01:39:24.993 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_manifest": 0, 2023-09-30T01:39:24.993 INFO:tasks.workunit.client.0.smithi186.stdout: "num_omap_bytes": 0, 2023-09-30T01:39:24.993 INFO:tasks.workunit.client.0.smithi186.stdout: "num_omap_keys": 0, 2023-09-30T01:39:24.993 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_repaired": 0 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: "store_stats": { 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: "total": 0, 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: "available": 0, 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: "internally_reserved": 0, 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: "allocated": 0, 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: "data_stored": 0, 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed": 0, 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed_allocated": 0, 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed_original": 0, 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: "omap_allocated": 0, 2023-09-30T01:39:24.994 INFO:tasks.workunit.client.0.smithi186.stdout: "internal_metadata": 0 2023-09-30T01:39:24.995 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.995 INFO:tasks.workunit.client.0.smithi186.stdout: "log_size": 2, 2023-09-30T01:39:24.995 INFO:tasks.workunit.client.0.smithi186.stdout: "ondisk_log_size": 2, 2023-09-30T01:39:24.995 INFO:tasks.workunit.client.0.smithi186.stdout: "up": 4, 2023-09-30T01:39:24.995 INFO:tasks.workunit.client.0.smithi186.stdout: "acting": 4, 2023-09-30T01:39:24.995 INFO:tasks.workunit.client.0.smithi186.stdout: "num_store_stats": 0 2023-09-30T01:39:24.995 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.995 INFO:tasks.workunit.client.0.smithi186.stdout: "osd_sum": { 2023-09-30T01:39:24.995 INFO:tasks.workunit.client.0.smithi186.stdout: "up_from": 0, 2023-09-30T01:39:24.995 INFO:tasks.workunit.client.0.smithi186.stdout: "seq": 0, 2023-09-30T01:39:24.995 INFO:tasks.workunit.client.0.smithi186.stdout: "num_pgs": 4, 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "num_osds": 1, 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "num_per_pool_osds": 1, 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "num_per_pool_omap_osds": 1, 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "kb": 104857600, 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "kb_used": 19828, 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "kb_used_data": 108, 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "kb_used_omap": 0, 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "kb_used_meta": 19712, 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "kb_avail": 104837772, 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "statfs": { 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "total": 107374182400, 2023-09-30T01:39:24.996 INFO:tasks.workunit.client.0.smithi186.stdout: "available": 107353878528, 2023-09-30T01:39:24.997 INFO:tasks.workunit.client.0.smithi186.stdout: "internally_reserved": 0, 2023-09-30T01:39:24.997 INFO:tasks.workunit.client.0.smithi186.stdout: "allocated": 110592, 2023-09-30T01:39:24.997 INFO:tasks.workunit.client.0.smithi186.stdout: "data_stored": 22762, 2023-09-30T01:39:24.997 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed": 0, 2023-09-30T01:39:24.997 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed_allocated": 0, 2023-09-30T01:39:24.997 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed_original": 0, 2023-09-30T01:39:24.997 INFO:tasks.workunit.client.0.smithi186.stdout: "omap_allocated": 0, 2023-09-30T01:39:24.997 INFO:tasks.workunit.client.0.smithi186.stdout: "internal_metadata": 20185088 2023-09-30T01:39:24.997 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.997 INFO:tasks.workunit.client.0.smithi186.stdout: "hb_peers": [], 2023-09-30T01:39:24.997 INFO:tasks.workunit.client.0.smithi186.stdout: "snap_trim_queue_len": 0, 2023-09-30T01:39:24.998 INFO:tasks.workunit.client.0.smithi186.stdout: "num_snap_trimming": 0, 2023-09-30T01:39:24.998 INFO:tasks.workunit.client.0.smithi186.stdout: "num_shards_repaired": 0, 2023-09-30T01:39:24.998 INFO:tasks.workunit.client.0.smithi186.stdout: "op_queue_age_hist": { 2023-09-30T01:39:24.998 INFO:tasks.workunit.client.0.smithi186.stdout: "histogram": [], 2023-09-30T01:39:24.998 INFO:tasks.workunit.client.0.smithi186.stdout: "upper_bound": 1 2023-09-30T01:39:24.998 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.998 INFO:tasks.workunit.client.0.smithi186.stdout: "perf_stat": { 2023-09-30T01:39:24.998 INFO:tasks.workunit.client.0.smithi186.stdout: "commit_latency_ms": 0, 2023-09-30T01:39:24.998 INFO:tasks.workunit.client.0.smithi186.stdout: "apply_latency_ms": 0, 2023-09-30T01:39:24.998 INFO:tasks.workunit.client.0.smithi186.stdout: "commit_latency_ns": 0, 2023-09-30T01:39:24.998 INFO:tasks.workunit.client.0.smithi186.stdout: "apply_latency_ns": 0 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: "alerts": [], 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: "network_ping_times": [] 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: "osd_sum_by_class": { 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: "hdd": { 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: "up_from": 0, 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: "seq": 0, 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: "num_pgs": 4, 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: "num_osds": 1, 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: "num_per_pool_osds": 1, 2023-09-30T01:39:24.999 INFO:tasks.workunit.client.0.smithi186.stdout: "num_per_pool_omap_osds": 1, 2023-09-30T01:39:25.000 INFO:tasks.workunit.client.0.smithi186.stdout: "kb": 104857600, 2023-09-30T01:39:25.000 INFO:tasks.workunit.client.0.smithi186.stdout: "kb_used": 19828, 2023-09-30T01:39:25.000 INFO:tasks.workunit.client.0.smithi186.stdout: "kb_used_data": 108, 2023-09-30T01:39:25.000 INFO:tasks.workunit.client.0.smithi186.stdout: "kb_used_omap": 0, 2023-09-30T01:39:25.000 INFO:tasks.workunit.client.0.smithi186.stdout: "kb_used_meta": 19712, 2023-09-30T01:39:25.000 INFO:tasks.workunit.client.0.smithi186.stdout: "kb_avail": 104837772, 2023-09-30T01:39:25.000 INFO:tasks.workunit.client.0.smithi186.stdout: "statfs": { 2023-09-30T01:39:25.000 INFO:tasks.workunit.client.0.smithi186.stdout: "total": 107374182400, 2023-09-30T01:39:25.000 INFO:tasks.workunit.client.0.smithi186.stdout: "available": 107353878528, 2023-09-30T01:39:25.000 INFO:tasks.workunit.client.0.smithi186.stdout: "internally_reserved": 0, 2023-09-30T01:39:25.001 INFO:tasks.workunit.client.0.smithi186.stdout: "allocated": 110592, 2023-09-30T01:39:25.001 INFO:tasks.workunit.client.0.smithi186.stdout: "data_stored": 22762, 2023-09-30T01:39:25.001 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed": 0, 2023-09-30T01:39:25.001 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed_allocated": 0, 2023-09-30T01:39:25.001 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed_original": 0, 2023-09-30T01:39:25.001 INFO:tasks.workunit.client.0.smithi186.stdout: "omap_allocated": 0, 2023-09-30T01:39:25.001 INFO:tasks.workunit.client.0.smithi186.stdout: "internal_metadata": 20185088 2023-09-30T01:39:25.001 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:25.001 INFO:tasks.workunit.client.0.smithi186.stdout: "hb_peers": [], 2023-09-30T01:39:25.001 INFO:tasks.workunit.client.0.smithi186.stdout: "snap_trim_queue_len": 0, 2023-09-30T01:39:25.001 INFO:tasks.workunit.client.0.smithi186.stdout: "num_snap_trimming": 0, 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: "num_shards_repaired": 0, 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: "op_queue_age_hist": { 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: "histogram": [], 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: "upper_bound": 1 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: "perf_stat": { 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: "commit_latency_ms": 0, 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: "apply_latency_ms": 0, 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: "commit_latency_ns": 0, 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: "apply_latency_ns": 0 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:25.002 INFO:tasks.workunit.client.0.smithi186.stdout: "alerts": [], 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: "network_ping_times": [] 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: "pool_stats": [ 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: "poolid": 1, 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: "num_pg": 4, 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: "stat_sum": { 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: "num_bytes": 19, 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects": 1, 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: "num_object_clones": 0, 2023-09-30T01:39:25.003 INFO:tasks.workunit.client.0.smithi186.stdout: "num_object_copies": 2, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_missing_on_primary": 0, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_missing": 0, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_degraded": 1, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_misplaced": 0, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_unfound": 0, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_dirty": 1, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_whiteouts": 0, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_read": 0, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_read_kb": 0, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_write": 2, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_write_kb": 2, 2023-09-30T01:39:25.004 INFO:tasks.workunit.client.0.smithi186.stdout: "num_scrub_errors": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_shallow_scrub_errors": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_deep_scrub_errors": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_recovered": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_bytes_recovered": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_keys_recovered": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_omap": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_hit_set_archive": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_bytes_hit_set_archive": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_flush": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_flush_kb": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_evict": 0, 2023-09-30T01:39:25.005 INFO:tasks.workunit.client.0.smithi186.stdout: "num_evict_kb": 0, 2023-09-30T01:39:25.006 INFO:tasks.workunit.client.0.smithi186.stdout: "num_promote": 0, 2023-09-30T01:39:25.006 INFO:tasks.workunit.client.0.smithi186.stdout: "num_flush_mode_high": 0, 2023-09-30T01:39:25.006 INFO:tasks.workunit.client.0.smithi186.stdout: "num_flush_mode_low": 0, 2023-09-30T01:39:25.006 INFO:tasks.workunit.client.0.smithi186.stdout: "num_evict_mode_some": 0, 2023-09-30T01:39:25.006 INFO:tasks.workunit.client.0.smithi186.stdout: "num_evict_mode_full": 0, 2023-09-30T01:39:25.006 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_pinned": 0, 2023-09-30T01:39:25.006 INFO:tasks.workunit.client.0.smithi186.stdout: "num_legacy_snapsets": 0, 2023-09-30T01:39:25.006 INFO:tasks.workunit.client.0.smithi186.stdout: "num_large_omap_objects": 0, 2023-09-30T01:39:25.006 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_manifest": 0, 2023-09-30T01:39:25.006 INFO:tasks.workunit.client.0.smithi186.stdout: "num_omap_bytes": 0, 2023-09-30T01:39:25.006 INFO:tasks.workunit.client.0.smithi186.stdout: "num_omap_keys": 0, 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: "num_objects_repaired": 0 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: "store_stats": { 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: "total": 0, 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: "available": 0, 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: "internally_reserved": 0, 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: "allocated": 4096, 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: "data_stored": 19, 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed": 0, 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed_allocated": 0, 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: "data_compressed_original": 0, 2023-09-30T01:39:25.007 INFO:tasks.workunit.client.0.smithi186.stdout: "omap_allocated": 0, 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: "internal_metadata": 0 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: "log_size": 2, 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: "ondisk_log_size": 2, 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: "up": 4, 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: "acting": 4, 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: "num_store_stats": 1 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: "osd_stats": [ 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:25.008 INFO:tasks.workunit.client.0.smithi186.stdout: "osd": 0, 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: "seq": 21474836482 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: "num_pg_by_state": [ 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: "state": "active+undersized+degraded", 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: "num": 1 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: "state": "active+undersized", 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: "num": 3 2023-09-30T01:39:25.009 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: "num_pg_by_osd": [ 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: "osd": 0, 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: "num_primary_pg": 4, 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: "num_acting_pg": 4, 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: "num_up_not_acting_pg": 0 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: "purged_snaps": [ 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: { 2023-09-30T01:39:25.010 INFO:tasks.workunit.client.0.smithi186.stdout: "pool": 1, 2023-09-30T01:39:25.011 INFO:tasks.workunit.client.0.smithi186.stdout: "purged_snaps": {} 2023-09-30T01:39:25.011 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:25.011 INFO:tasks.workunit.client.0.smithi186.stdout: ], 2023-09-30T01:39:25.011 INFO:tasks.workunit.client.0.smithi186.stdout: "servicemap": { 2023-09-30T01:39:25.011 INFO:tasks.workunit.client.0.smithi186.stdout: "epoch": 1, 2023-09-30T01:39:25.011 INFO:tasks.workunit.client.0.smithi186.stdout: "modified": "2023-09-30T01:38:37.325969+0000", 2023-09-30T01:39:25.011 INFO:tasks.workunit.client.0.smithi186.stdout: "services": {} 2023-09-30T01:39:25.011 INFO:tasks.workunit.client.0.smithi186.stdout: }, 2023-09-30T01:39:25.011 INFO:tasks.workunit.client.0.smithi186.stdout: "paxos": { 2023-09-30T01:39:25.011 INFO:tasks.workunit.client.0.smithi186.stdout: "first_committed": 1, 2023-09-30T01:39:25.011 INFO:tasks.workunit.client.0.smithi186.stdout: "last_committed": 111, 2023-09-30T01:39:25.012 INFO:tasks.workunit.client.0.smithi186.stdout: "last_pn": 0, 2023-09-30T01:39:25.012 INFO:tasks.workunit.client.0.smithi186.stdout: "accepted_pn": 0 2023-09-30T01:39:25.012 INFO:tasks.workunit.client.0.smithi186.stdout: } 2023-09-30T01:39:25.012 INFO:tasks.workunit.client.0.smithi186.stdout:} 2023-09-30T01:39:25.014 INFO:tasks.workunit.client.0.smithi186.stderr:report 963613784 2023-09-30T01:39:25.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2023-09-30T01:39:25.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: test_wait_for_clean: run_osd td/ceph-helpers 1 2023-09-30T01:39:25.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:39:25.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:39:25.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-09-30T01:39:25.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:39:25.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2023-09-30T01:39:25.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:39:25.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:39:25.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:39:25.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:39:25.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2023-09-30T01:39:25.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2023-09-30T01:39:25.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:39:25.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:39:25.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:39:25.015 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:39:25.016 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:39:25.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:39:25.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:39:25.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:39:25.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:39:25.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:39:25.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:39:25.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2023-09-30T01:39:25.017 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:39:25.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=934fb153-14b4-490c-b2bd-ecec220968e2 2023-09-30T01:39:25.017 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 934fb153-14b4-490c-b2bd-ecec220968e2' 2023-09-30T01:39:25.018 INFO:tasks.workunit.client.0.smithi186.stdout:add osd1 934fb153-14b4-490c-b2bd-ecec220968e2 2023-09-30T01:39:25.018 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:39:25.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMfBdls7EfOhAACJBlMd/brtI6nDF3NmvP2w== 2023-09-30T01:39:25.018 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMfBdls7EfOhAACJBlMd/brtI6nDF3NmvP2w=="}' 2023-09-30T01:39:25.019 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 934fb153-14b4-490c-b2bd-ecec220968e2 -i td/ceph-helpers/1/new.json 2023-09-30T01:39:25.535 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:39:25.552 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2023-09-30T01:39:25.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMfBdls7EfOhAACJBlMd/brtI6nDF3NmvP2w== --osd-uuid 934fb153-14b4-490c-b2bd-ecec220968e2 2023-09-30T01:39:25.584 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:25.581+0000 7fcdbf1ad3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:39:25.594 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:25.589+0000 7fcdbf1ad3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:39:25.596 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:25.593+0000 7fcdbf1ad3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:39:25.596 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:25.593+0000 7fcdbf1ad3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:39:25.596 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:25.593+0000 7fcdbf1ad3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:39:25.597 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:25.593+0000 7fcdbf1ad3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:39:25.687 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:25.681+0000 7fcdbf1ad3c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2023-09-30T01:39:31.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2023-09-30T01:39:31.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:39:31.760 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd1 key to auth repository 2023-09-30T01:39:31.760 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-09-30T01:39:31.761 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:39:32.333 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:39:32.333 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-09-30T01:39:32.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:39:32.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:39:32.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:39:32.334 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:39:32.362 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:32.357+0000 7f166c2833c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:39:32.365 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:32.361+0000 7f166c2833c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:39:32.368 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:32.361+0000 7f166c2833c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:39:32.794 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-09-30T01:39:32.794 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:39:32.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:39:32.795 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:39:32.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:39:32.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:39:32.795 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:32.796 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:39:32.796 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:32.796 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:32.926 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:32.921+0000 7f166c2833c0 -1 Falling back to public interface 2023-09-30T01:39:33.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:34.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:34.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:34.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:39:34.239 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:39:34.240 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:34.240 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:34.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:35.533 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:39:35.529+0000 7f166c2833c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:39:35.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:35.679 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:35.680 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:39:35.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:39:35.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:35.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:36.224 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:37.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:37.226 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:37.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:39:37.227 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:39:37.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:37.227 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:37.678 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:38.680 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:38.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:38.681 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:39:38.681 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:39:38.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:38.682 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:39.122 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:40.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:40.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:40.124 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:39:40.125 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:39:40.125 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:40.125 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:40.786 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:41.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:41.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:41.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:39:41.789 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:39:41.789 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:41.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:42.321 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:43.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:43.323 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:43.323 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:39:43.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:39:43.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:43.324 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:43.743 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:44.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:44.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:44.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:39:44.746 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:39:44.746 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:44.747 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:45.196 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:46.198 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:46.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:46.199 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:39:46.199 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:39:46.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:46.200 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:46.717 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:47.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:47.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:47.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:39:47.719 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:39:47.720 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:47.720 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:48.150 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:49.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:49.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:49.152 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:39:49.152 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:39:49.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:49.153 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:49.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:50.758 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:50.759 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:50.759 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:39:50.759 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:39:50.760 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:50.760 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:51.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:52.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:52.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:52.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:39:52.561 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:39:52.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:52.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:53.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:54.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:54.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:54.538 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:39:54.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:39:54.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:54.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:55.190 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:56.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:56.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:56.192 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:39:56.193 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:39:56.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:56.193 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:56.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:57.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:57.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:57.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:39:57.989 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:39:57.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:57.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:39:58.899 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:39:59.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:39:59.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:39:59.901 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2023-09-30T01:39:59.901 INFO:tasks.workunit.client.0.smithi186.stdout:17 2023-09-30T01:39:59.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:39:59.902 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:40:00.626 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 16 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/152924,v1:127.0.0.1:6811/152924] [v2:127.0.0.1:6812/152924,v1:127.0.0.1:6813/152924] exists,up 934fb153-14b4-490c-b2bd-ecec220968e2 2023-09-30T01:40:00.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:40:00.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:40:00.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:40:00.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1686: test_wait_for_clean: wait_for_clean 2023-09-30T01:40:00.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-09-30T01:40:00.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-09-30T01:40:00.627 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-09-30T01:40:00.628 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-09-30T01:40:00.628 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-09-30T01:40:00.628 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:40:00.628 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:40:00.629 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:40:00.629 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:40:00.629 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:40:00.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-09-30T01:40:00.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-09-30T01:40:00.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-09-30T01:40:00.830 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: flush_pg_stats: local timeout=300 2023-09-30T01:40:00.831 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ceph osd ls 2023-09-30T01:40:01.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: flush_pg_stats: ids='0 2023-09-30T01:40:01.330 INFO:tasks.workunit.client.0.smithi186.stderr:1' 2023-09-30T01:40:01.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: flush_pg_stats: seqs= 2023-09-30T01:40:01.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:40:01.331 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-09-30T01:40:01.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=21474836492 2023-09-30T01:40:01.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 21474836492 2023-09-30T01:40:01.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836492' 2023-09-30T01:40:01.498 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: flush_pg_stats: for osd in $ids 2023-09-30T01:40:01.498 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-09-30T01:40:01.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: flush_pg_stats: seq=68719476751 2023-09-30T01:40:01.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: flush_pg_stats: test -z 68719476751 2023-09-30T01:40:01.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: flush_pg_stats: seqs=' 0-21474836492 1-68719476751' 2023-09-30T01:40:01.666 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:40:01.666 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 0-21474836492 2023-09-30T01:40:01.667 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:40:01.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=0 2023-09-30T01:40:01.670 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 0-21474836492 2023-09-30T01:40:01.670 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:40:01.671 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=21474836492 2023-09-30T01:40:01.671 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2023-09-30T01:40:01.671 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.0 seq 21474836492 2023-09-30T01:40:01.672 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:40:02.125 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836490 -lt 21474836492 2023-09-30T01:40:02.125 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: flush_pg_stats: sleep 1 2023-09-30T01:40:03.127 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: flush_pg_stats: '[' 300 -eq 0 ']' 2023-09-30T01:40:03.127 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 0 2023-09-30T01:40:03.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 21474836492 -lt 21474836492 2023-09-30T01:40:03.582 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: flush_pg_stats: for s in $seqs 2023-09-30T01:40:03.583 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: echo 1-68719476751 2023-09-30T01:40:03.583 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: cut -d - -f 1 2023-09-30T01:40:03.585 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: flush_pg_stats: osd=1 2023-09-30T01:40:03.586 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: echo 1-68719476751 2023-09-30T01:40:03.586 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: cut -d - -f 2 2023-09-30T01:40:03.587 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: flush_pg_stats: seq=68719476751 2023-09-30T01:40:03.587 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: flush_pg_stats: echo 'waiting osd.1 seq 68719476751' 2023-09-30T01:40:03.588 INFO:tasks.workunit.client.0.smithi186.stdout:waiting osd.1 seq 68719476751 2023-09-30T01:40:03.588 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: ceph osd last-stat-seq 1 2023-09-30T01:40:04.036 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: flush_pg_stats: test 68719476751 -lt 68719476751 2023-09-30T01:40:04.037 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-09-30T01:40:04.038 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:40:04.038 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:40:04.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2023-09-30T01:40:04.619 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-09-30T01:40:04.620 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-09-30T01:40:04.620 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-09-30T01:40:04.621 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-09-30T01:40:04.621 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-09-30T01:40:04.621 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-09-30T01:40:04.621 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-09-30T01:40:05.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2023-09-30T01:40:05.043 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-09-30T01:40:05.043 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-09-30T01:40:05.044 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-09-30T01:40:05.650 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2023-09-30T01:40:05.650 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-09-30T01:40:05.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-09-30T01:40:05.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: test_wait_for_clean: teardown td/ceph-helpers 2023-09-30T01:40:05.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:40:05.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:40:05.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:40:05.652 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:40:05.652 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:40:05.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:40:05.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:40:05.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:40:05.766 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:40:05.767 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:40:05.768 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:40:05.769 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:40:05.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:40:05.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:40:05.771 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:40:05.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:40:05.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:40:05.772 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:40:05.773 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:40:05.774 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:40:05.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:40:05.776 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:40:05.787 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:40:05.787 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:40:05.788 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:40:05.788 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:40:05.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:40:05.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:40:05.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: run_tests: for func in $funcs 2023-09-30T01:40:05.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: run_tests: test_wait_for_health_ok td/ceph-helpers 2023-09-30T01:40:05.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1783: test_wait_for_health_ok: local dir=td/ceph-helpers 2023-09-30T01:40:05.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: test_wait_for_health_ok: setup td/ceph-helpers 2023-09-30T01:40:05.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2023-09-30T01:40:05.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2023-09-30T01:40:05.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:40:05.790 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-09-30T01:40:05.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:40:05.791 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:40:05.791 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:40:05.791 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:40:05.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:40:05.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:40:05.794 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:40:05.794 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:40:05.796 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:40:05.796 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:40:05.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:40:05.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:40:05.798 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:40:05.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:40:05.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:40:05.800 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:40:05.801 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:40:05.802 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:40:05.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-09-30T01:40:05.804 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:40:05.806 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:40:05.806 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:40:05.806 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:40:05.806 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:40:05.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:40:05.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:40:05.808 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2023-09-30T01:40:05.811 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-09-30T01:40:05.811 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:40:05.811 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:40:05.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76804 2023-09-30T01:40:05.813 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-09-30T01:40:05.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-09-30T01:40:05.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-09-30T01:40:05.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2023-09-30T01:40:05.815 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: test_wait_for_health_ok: run_mon td/ceph-helpers a --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2023-09-30T01:40:05.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2023-09-30T01:40:05.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-09-30T01:40:05.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-09-30T01:40:05.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-09-30T01:40:05.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2023-09-30T01:40:05.816 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2023-09-30T01:40:06.178 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-09-30T01:40:06.178 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:40:06.178 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:40:06.179 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:40:06.179 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:40:06.179 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:40:06.179 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:40:06.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2023-09-30T01:40:06.506 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-09-30T01:40:06.508 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-09-30T01:40:06.508 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:40:06.508 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:40:06.508 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-09-30T01:40:06.508 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-09-30T01:40:06.509 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:40:06.509 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:40:06.509 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:40:06.510 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:40:06.510 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:40:06.510 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:40:06.510 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:40:06.511 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:40:06.511 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get fsid 2023-09-30T01:40:06.616 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-09-30T01:40:06.616 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-09-30T01:40:06.616 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-09-30T01:40:06.616 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-09-30T01:40:06.616 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-09-30T01:40:06.617 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-09-30T01:40:06.617 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-09-30T01:40:06.617 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-09-30T01:40:06.618 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-09-30T01:40:06.618 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:40:06.618 INFO:tasks.workunit.client.0.smithi186.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:40:06.618 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76804/ceph-mon.a.asok 2023-09-30T01:40:06.619 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-09-30T01:40:06.619 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.76804/ceph-mon.a.asok config get mon_host 2023-09-30T01:40:06.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1787: test_wait_for_health_ok: run_mgr td/ceph-helpers x --mon_pg_warn_min_per_osd=0 2023-09-30T01:40:06.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2023-09-30T01:40:06.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-09-30T01:40:06.703 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-09-30T01:40:06.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-09-30T01:40:06.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2023-09-30T01:40:06.704 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-09-30T01:40:07.613 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-09-30T01:40:07.613 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:40:07.613 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:40:07.614 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:40:07.614 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:40:07.614 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:40:07.614 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:40:07.615 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-09-30T01:40:07.617 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_pg_warn_min_per_osd=0 2023-09-30T01:40:07.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1789: test_wait_for_health_ok: run_osd td/ceph-helpers 0 2023-09-30T01:40:07.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:40:07.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:40:07.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-09-30T01:40:07.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:40:07.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:40:07.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:40:07.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:40:07.654 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:40:07.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:40:07.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:40:07.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:40:07.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:40:07.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:40:07.655 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:40:07.655 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:40:07.655 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:40:07.655 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:40:07.657 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:40:07.657 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:40:07.657 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:40:07.657 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:40:07.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:40:07.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:40:07.657 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:40:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:40:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:40:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:40:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:40:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:40:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:40:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:40:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:40:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:40:07.658 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:40:07.659 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:40:07.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=337ac10e-699d-4f56-9cb3-d90e7ee0975a 2023-09-30T01:40:07.659 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 337ac10e-699d-4f56-9cb3-d90e7ee0975a' 2023-09-30T01:40:07.659 INFO:tasks.workunit.client.0.smithi186.stdout:add osd0 337ac10e-699d-4f56-9cb3-d90e7ee0975a 2023-09-30T01:40:07.660 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:40:07.676 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3fBdl+oNJKBAAy9JqYQx0LMN1RqWYvWV4aQ== 2023-09-30T01:40:07.677 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3fBdl+oNJKBAAy9JqYQx0LMN1RqWYvWV4aQ=="}' 2023-09-30T01:40:07.677 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 337ac10e-699d-4f56-9cb3-d90e7ee0975a -i td/ceph-helpers/0/new.json 2023-09-30T01:40:08.040 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:40:08.053 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2023-09-30T01:40:08.055 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3fBdl+oNJKBAAy9JqYQx0LMN1RqWYvWV4aQ== --osd-uuid 337ac10e-699d-4f56-9cb3-d90e7ee0975a 2023-09-30T01:40:08.086 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:08.080+0000 7f93c010e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:08.088 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:08.084+0000 7f93c010e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:08.089 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:08.084+0000 7f93c010e3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:08.090 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:08.084+0000 7f93c010e3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:40:08.090 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:08.084+0000 7f93c010e3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:40:08.091 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:08.084+0000 7f93c010e3c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2023-09-30T01:40:08.170 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:08.164+0000 7f93c010e3c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2023-09-30T01:40:14.835 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2023-09-30T01:40:14.835 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:40:14.837 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd0 key to auth repository 2023-09-30T01:40:14.837 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-09-30T01:40:14.837 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:40:15.386 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-09-30T01:40:15.387 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:40:15.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:40:15.387 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:40:15.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:40:15.388 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:40:15.417 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:15.412+0000 7f601b4c83c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:15.419 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:15.416+0000 7f601b4c83c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:15.423 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:15.416+0000 7f601b4c83c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:15.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-09-30T01:40:15.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:40:15.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:40:15.769 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:40:15.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:40:15.770 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:15.770 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:40:15.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:40:15.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:15.771 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:15.969 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:15.964+0000 7f601b4c83c0 -1 Falling back to public interface 2023-09-30T01:40:16.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:17.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:17.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:17.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:40:17.218 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:40:17.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:17.218 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:17.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:18.301 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:18.296+0000 7f601b4c83c0 -1 osd.0 0 log_to_monitors true 2023-09-30T01:40:18.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:18.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:18.667 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:40:18.668 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:40:18.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:18.668 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:19.254 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:20.256 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:20.256 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:20.257 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:40:20.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:40:20.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:20.257 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:20.686 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:21.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:21.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:21.688 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:40:21.688 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:40:21.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:21.689 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:22.103 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:23.104 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:23.105 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:23.105 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:40:23.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:40:23.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:23.106 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:23.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:24.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:24.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:24.538 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:40:24.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:40:24.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:24.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:25.010 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:26.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:26.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:26.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:40:26.013 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:40:26.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:26.013 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:26.458 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:27.460 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:27.461 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:27.461 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:40:27.462 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:40:27.462 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:27.462 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:28.040 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:29.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:29.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:29.042 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:40:29.043 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:40:29.043 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:29.043 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:29.473 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:30.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:30.475 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:30.475 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:40:30.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:40:30.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:30.476 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:31.030 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:32.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:32.032 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:32.032 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:40:32.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:40:32.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:32.033 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:32.559 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:33.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:33.561 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:33.561 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:40:33.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:40:33.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:33.562 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:34.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:35.279 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:35.279 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:35.279 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:40:35.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:40:35.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:35.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:36.182 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:37.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:37.184 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:37.185 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:40:37.185 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:40:37.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:37.186 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:37.965 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:38.967 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:38.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:38.968 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:40:38.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:40:38.968 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:38.969 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:39.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:40.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:40.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:40.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:40:40.860 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:40:40.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:40.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:40:41.274 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/155339,v1:127.0.0.1:6803/155339] [v2:127.0.0.1:6804/155339,v1:127.0.0.1:6805/155339] exists,up 337ac10e-699d-4f56-9cb3-d90e7ee0975a 2023-09-30T01:40:41.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:40:41.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:40:41.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:40:41.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: test_wait_for_health_ok: run_osd td/ceph-helpers 1 2023-09-30T01:40:41.276 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:40:41.277 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:40:41.278 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:40:41.278 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:40:41.278 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:40:41.278 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:40:41.279 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:40:41.280 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:40:41.280 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:40:41.280 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:40:41.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:40:41.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:40:41.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:40:41.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:40:41.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:40:41.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:40:41.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:40:41.280 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:40:41.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:40:41.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:40:41.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:40:41.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:40:41.281 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2023-09-30T01:40:41.281 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:40:41.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=925f5ab8-5f7f-4498-8ee2-3c3aa6cdec69 2023-09-30T01:40:41.282 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 925f5ab8-5f7f-4498-8ee2-3c3aa6cdec69' 2023-09-30T01:40:41.282 INFO:tasks.workunit.client.0.smithi186.stdout:add osd1 925f5ab8-5f7f-4498-8ee2-3c3aa6cdec69 2023-09-30T01:40:41.282 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:40:41.302 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZfBdlxOIEEhAA6JvJ/X+neS/KcWX5rJjLXA== 2023-09-30T01:40:41.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZfBdlxOIEEhAA6JvJ/X+neS/KcWX5rJjLXA=="}' 2023-09-30T01:40:41.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 925f5ab8-5f7f-4498-8ee2-3c3aa6cdec69 -i td/ceph-helpers/1/new.json 2023-09-30T01:40:41.906 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:40:41.921 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2023-09-30T01:40:41.922 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZfBdlxOIEEhAA6JvJ/X+neS/KcWX5rJjLXA== --osd-uuid 925f5ab8-5f7f-4498-8ee2-3c3aa6cdec69 2023-09-30T01:40:41.956 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:41.952+0000 7fc48901a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:41.959 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:41.952+0000 7fc48901a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:41.961 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:41.956+0000 7fc48901a3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:41.962 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:41.956+0000 7fc48901a3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:40:41.962 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:41.956+0000 7fc48901a3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:40:41.962 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:41.956+0000 7fc48901a3c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2023-09-30T01:40:42.116 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:42.116+0000 7fc48901a3c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2023-09-30T01:40:48.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2023-09-30T01:40:48.449 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:40:48.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-09-30T01:40:48.451 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd1 key to auth repository 2023-09-30T01:40:48.451 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:40:48.998 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:40:48.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-09-30T01:40:48.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:40:48.999 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:40:49.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:40:49.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:40:49.029 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:49.028+0000 7fc91fb023c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:49.056 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:49.056+0000 7fc91fb023c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:49.057 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:49.056+0000 7fc91fb023c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:40:49.456 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-09-30T01:40:49.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:40:49.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:40:49.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:40:49.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:40:49.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:49.458 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:40:49.458 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:40:49.458 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:49.459 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:40:49.617 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:49.616+0000 7fc91fb023c0 -1 Falling back to public interface 2023-09-30T01:40:49.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:50.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:50.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:50.861 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:40:50.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:40:50.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:50.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:40:51.264 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:52.245 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:40:52.244+0000 7fc91fb023c0 -1 osd.1 0 log_to_monitors true 2023-09-30T01:40:52.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:52.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:52.266 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:40:52.266 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:40:52.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:52.267 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:40:52.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:53.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:53.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:53.731 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:40:53.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:40:53.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:53.731 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:40:54.303 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:55.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:55.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:55.305 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:40:55.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:40:55.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:55.306 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:40:55.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:56.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:56.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:56.722 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:40:56.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:40:56.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:56.723 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:40:57.272 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:58.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:58.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:58.274 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:40:58.275 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:40:58.275 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:58.275 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:40:58.735 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:40:59.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:40:59.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:40:59.737 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:40:59.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:40:59.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:40:59.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:41:00.305 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:01.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:01.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:01.307 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:41:01.308 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:41:01.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:01.308 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:41:01.727 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:02.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:02.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:02.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:41:02.730 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:41:02.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:02.730 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:41:03.160 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:04.161 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:04.162 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:04.162 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:41:04.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:41:04.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:04.163 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:41:04.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:05.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:05.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:05.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:41:05.960 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:41:05.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:05.961 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:41:06.946 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:07.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:07.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:07.948 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:41:07.948 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:41:07.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:07.949 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:41:08.745 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:09.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:09.748 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:09.748 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:41:09.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:41:09.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:09.749 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:41:10.567 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:11.569 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:11.569 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:11.569 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:41:11.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:41:11.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:11.570 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:41:12.374 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:13.376 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:13.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:13.377 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:41:13.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:41:13.377 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:13.378 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:41:14.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:15.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:15.167 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:15.167 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:41:15.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:41:15.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:15.168 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:41:15.600 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/157089,v1:127.0.0.1:6811/157089] [v2:127.0.0.1:6812/157089,v1:127.0.0.1:6813/157089] exists,up 925f5ab8-5f7f-4498-8ee2-3c3aa6cdec69 2023-09-30T01:41:15.602 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:41:15.602 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:41:15.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:41:15.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: test_wait_for_health_ok: run_osd td/ceph-helpers 2 2023-09-30T01:41:15.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2023-09-30T01:41:15.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-09-30T01:41:15.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-09-30T01:41:15.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-09-30T01:41:15.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2023-09-30T01:41:15.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:41:15.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:41:15.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:41:15.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:41:15.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2023-09-30T01:41:15.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2023-09-30T01:41:15.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-09-30T01:41:15.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-09-30T01:41:15.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:41:15.604 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-09-30T01:41:15.604 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:41:15.604 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:41:15.604 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:41:15.604 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:41:15.605 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:41:15.606 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:41:15.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:41:15.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:41:15.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-09-30T01:41:15.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-09-30T01:41:15.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:41:15.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:41:15.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:41:15.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:41:15.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:41:15.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:41:15.607 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-09-30T01:41:15.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2023-09-30T01:41:15.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2023-09-30T01:41:15.608 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-09-30T01:41:15.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a5386f0c-9e21-4393-abf6-d046a8a818f2 2023-09-30T01:41:15.608 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a5386f0c-9e21-4393-abf6-d046a8a818f2' 2023-09-30T01:41:15.609 INFO:tasks.workunit.client.0.smithi186.stdout:add osd2 a5386f0c-9e21-4393-abf6-d046a8a818f2 2023-09-30T01:41:15.609 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-09-30T01:41:15.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7fBdlVpV6JRAAyB8OEErMlYal8nUZJ74x8g== 2023-09-30T01:41:15.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7fBdlVpV6JRAAyB8OEErMlYal8nUZJ74x8g=="}' 2023-09-30T01:41:15.629 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a5386f0c-9e21-4393-abf6-d046a8a818f2 -i td/ceph-helpers/2/new.json 2023-09-30T01:41:16.219 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:41:16.233 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2023-09-30T01:41:16.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7fBdlVpV6JRAAyB8OEErMlYal8nUZJ74x8g== --osd-uuid a5386f0c-9e21-4393-abf6-d046a8a818f2 2023-09-30T01:41:16.267 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:16.263+0000 7f42fba593c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:41:16.270 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:16.267+0000 7f42fba593c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:41:16.272 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:16.271+0000 7f42fba593c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:41:16.273 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:16.271+0000 7f42fba593c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2023-09-30T01:41:16.273 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:16.271+0000 7f42fba593c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2023-09-30T01:41:16.273 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:16.271+0000 7f42fba593c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2023-09-30T01:41:16.357 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:16.355+0000 7f42fba593c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2023-09-30T01:41:22.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2023-09-30T01:41:22.674 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-09-30T01:41:22.676 INFO:tasks.workunit.client.0.smithi186.stdout:adding osd2 key to auth repository 2023-09-30T01:41:22.676 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-09-30T01:41:22.676 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-09-30T01:41:23.205 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.2 2023-09-30T01:41:23.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-09-30T01:41:23.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:41:23.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-09-30T01:41:23.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-09-30T01:41:23.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-09-30T01:41:23.237 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:23.235+0000 7f695971f3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:41:23.240 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:23.239+0000 7f695971f3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:41:23.241 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:23.239+0000 7f695971f3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:41:23.651 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-09-30T01:41:23.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:41:23.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-09-30T01:41:23.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:41:23.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:41:23.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:23.652 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:41:23.652 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:41:23.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:23.653 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:23.777 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:23.775+0000 7f695971f3c0 -1 Falling back to public interface 2023-09-30T01:41:24.022 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:25.024 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:25.024 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:25.024 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:41:25.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:41:25.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:25.025 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:25.426 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:26.348 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:26.347+0000 7f695971f3c0 -1 osd.2 0 log_to_monitors true 2023-09-30T01:41:26.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:26.428 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:26.429 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:41:26.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:41:26.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:26.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:27.109 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:28.113 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:28.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:28.114 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:41:28.114 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:41:28.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:28.115 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:28.641 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:29.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:29.643 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:29.644 INFO:tasks.workunit.client.0.smithi186.stdout:4 2023-09-30T01:41:29.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2023-09-30T01:41:29.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:29.645 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:30.052 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:31.054 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:31.055 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:31.055 INFO:tasks.workunit.client.0.smithi186.stdout:5 2023-09-30T01:41:31.055 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2023-09-30T01:41:31.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:31.056 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:31.505 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:32.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:32.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:32.507 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2023-09-30T01:41:32.508 INFO:tasks.workunit.client.0.smithi186.stdout:6 2023-09-30T01:41:32.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:32.508 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:33.012 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:34.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:34.014 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:34.014 INFO:tasks.workunit.client.0.smithi186.stdout:7 2023-09-30T01:41:34.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2023-09-30T01:41:34.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:34.015 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:34.427 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:35.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:35.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:35.429 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2023-09-30T01:41:35.430 INFO:tasks.workunit.client.0.smithi186.stdout:8 2023-09-30T01:41:35.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:35.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:35.986 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:36.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:36.988 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:36.988 INFO:tasks.workunit.client.0.smithi186.stdout:9 2023-09-30T01:41:36.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2023-09-30T01:41:36.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:36.989 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:37.523 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:38.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:38.525 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:38.525 INFO:tasks.workunit.client.0.smithi186.stdout:10 2023-09-30T01:41:38.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2023-09-30T01:41:38.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:38.526 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:39.061 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:40.062 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:40.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:40.063 INFO:tasks.workunit.client.0.smithi186.stdout:11 2023-09-30T01:41:40.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2023-09-30T01:41:40.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:40.064 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:40.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:41.914 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:41.914 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:41.915 INFO:tasks.workunit.client.0.smithi186.stdout:12 2023-09-30T01:41:41.915 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2023-09-30T01:41:41.915 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:41.915 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:42.612 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:43.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:43.614 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:43.614 INFO:tasks.workunit.client.0.smithi186.stdout:13 2023-09-30T01:41:43.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2023-09-30T01:41:43.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:43.615 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:44.536 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:45.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:45.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:45.538 INFO:tasks.workunit.client.0.smithi186.stdout:14 2023-09-30T01:41:45.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2023-09-30T01:41:45.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:45.539 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:46.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:47.344 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:47.344 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:47.344 INFO:tasks.workunit.client.0.smithi186.stdout:15 2023-09-30T01:41:47.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2023-09-30T01:41:47.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:47.345 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:48.239 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:49.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:49.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:49.241 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2023-09-30T01:41:49.241 INFO:tasks.workunit.client.0.smithi186.stdout:16 2023-09-30T01:41:49.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:49.242 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:41:49.662 INFO:tasks.workunit.client.0.smithi186.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/158838,v1:127.0.0.1:6819/158838] [v2:127.0.0.1:6820/158838,v1:127.0.0.1:6821/158838] exists,up a5386f0c-9e21-4393-abf6-d046a8a818f2 2023-09-30T01:41:49.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:41:49.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:41:49.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:41:49.663 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1792: test_wait_for_health_ok: kill_daemons td/ceph-helpers TERM osd 2023-09-30T01:41:49.664 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:41:49.664 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:41:49.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:41:49.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:41:49.665 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:41:49.977 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:41:49.978 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1793: test_wait_for_health_ok: ceph osd down 0 2023-09-30T01:41:50.447 INFO:tasks.workunit.client.0.smithi186.stderr:osd.0 is already down. 2023-09-30T01:41:50.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: test_wait_for_health_ok: TIMEOUT=1 2023-09-30T01:41:50.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1795: test_wait_for_health_ok: wait_for_health_ok 2023-09-30T01:41:50.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1779: wait_for_health_ok: wait_for_health HEALTH_OK 2023-09-30T01:41:50.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: wait_for_health: local grepstr=HEALTH_OK 2023-09-30T01:41:50.458 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: delays=($(get_timeout_delays $TIMEOUT .1)) 2023-09-30T01:41:50.458 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: get_timeout_delays 1 .1 2023-09-30T01:41:50.459 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:41:50.460 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:41:50.460 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:41:50.460 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:41:50.460 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:41:50.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: local -a delays 2023-09-30T01:41:50.519 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: wait_for_health: local -i loop=0 2023-09-30T01:41:50.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:41:50.520 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:41:51.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 0 >= 4 )) 2023-09-30T01:41:51.063 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.1 2023-09-30T01:41:51.164 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:41:51.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:41:51.165 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:41:51.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 1 >= 4 )) 2023-09-30T01:41:51.670 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.2 2023-09-30T01:41:51.871 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:41:51.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:41:51.872 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:41:52.390 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 2 >= 4 )) 2023-09-30T01:41:52.390 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.4 2023-09-30T01:41:52.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:41:52.792 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:41:52.793 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:41:53.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 3 >= 4 )) 2023-09-30T01:41:53.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.3 2023-09-30T01:41:53.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:41:53.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:41:53.644 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:41:54.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 4 >= 4 )) 2023-09-30T01:41:54.175 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1764: wait_for_health: ceph health detail 2023-09-30T01:41:54.714 INFO:tasks.workunit.client.0.smithi186.stdout:HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'; 3 osds down; 1 host (3 osds) down; 1 root (3 osds) down 2023-09-30T01:41:54.714 INFO:tasks.workunit.client.0.smithi186.stdout:[ERR] MGR_MODULE_ERROR: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:41:54.714 INFO:tasks.workunit.client.0.smithi186.stdout: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:41:54.714 INFO:tasks.workunit.client.0.smithi186.stdout:[WRN] OSD_DOWN: 3 osds down 2023-09-30T01:41:54.714 INFO:tasks.workunit.client.0.smithi186.stdout: osd.0 (root=default,host=smithi186) is down 2023-09-30T01:41:54.715 INFO:tasks.workunit.client.0.smithi186.stdout: osd.1 (root=default,host=smithi186) is down 2023-09-30T01:41:54.715 INFO:tasks.workunit.client.0.smithi186.stdout: osd.2 (root=default,host=smithi186) is down 2023-09-30T01:41:54.715 INFO:tasks.workunit.client.0.smithi186.stdout:[WRN] OSD_HOST_DOWN: 1 host (3 osds) down 2023-09-30T01:41:54.715 INFO:tasks.workunit.client.0.smithi186.stdout: host smithi186 (root=default) (3 osds) is down 2023-09-30T01:41:54.715 INFO:tasks.workunit.client.0.smithi186.stdout:[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down 2023-09-30T01:41:54.715 INFO:tasks.workunit.client.0.smithi186.stdout: root default (3 osds) is down 2023-09-30T01:41:54.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1765: wait_for_health: return 1 2023-09-30T01:41:54.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1779: wait_for_health_ok: return 1 2023-09-30T01:41:54.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1797: test_wait_for_health_ok: activate_osd td/ceph-helpers 0 2023-09-30T01:41:54.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2023-09-30T01:41:54.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-09-30T01:41:54.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-09-30T01:41:54.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-09-30T01:41:54.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2023-09-30T01:41:54.732 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:41:54.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:41:54.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:41:54.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:41:54.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2023-09-30T01:41:54.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2023-09-30T01:41:54.733 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-09-30T01:41:54.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-09-30T01:41:54.734 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:41:54.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-09-30T01:41:54.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:41:54.734 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:41:54.736 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:41:54.736 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:41:54.736 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:41:54.736 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:41:54.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:41:54.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:41:54.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:41:54.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:41:54.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:41:54.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:41:54.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:41:54.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:41:54.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-09-30T01:41:54.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-09-30T01:41:54.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2023-09-30T01:41:54.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-09-30T01:41:54.737 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.0 2023-09-30T01:41:54.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:41:54.738 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2023-09-30T01:41:54.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-09-30T01:41:54.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-09-30T01:41:54.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-09-30T01:41:54.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-09-30T01:41:54.766 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:54.762+0000 7f4d50f6b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:41:54.769 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:54.766+0000 7f4d50f6b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:41:54.770 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:54.766+0000 7f4d50f6b3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:41:55.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-09-30T01:41:55.129 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:41:55.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-09-30T01:41:55.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:41:55.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:41:55.130 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:55.130 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:41:55.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:41:55.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:55.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:41:55.305 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:55.302+0000 7f4d50f6b3c0 -1 Falling back to public interface 2023-09-30T01:41:55.527 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:56.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:56.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:56.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:41:56.531 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:41:56.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:56.532 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:41:56.910 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:57.911 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:57.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:57.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:41:57.912 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:41:57.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:57.912 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:41:58.017 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:58.014+0000 7f4d50f6b3c0 -1 osd.0 14 log_to_monitors true 2023-09-30T01:41:58.294 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:41:59.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:41:59.296 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:41:59.297 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:41:59.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:41:59.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:41:59.297 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-09-30T01:41:59.715 INFO:tasks.workunit.client.0.smithi186.stdout:osd.0 up in weight 1 up_from 18 up_thru 0 down_at 16 last_clean_interval [5,14) [v2:127.0.0.1:6802/159990,v1:127.0.0.1:6803/159990] [v2:127.0.0.1:6804/159990,v1:127.0.0.1:6805/159990] exists,up 337ac10e-699d-4f56-9cb3-d90e7ee0975a 2023-09-30T01:41:59.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:41:59.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:41:59.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:41:59.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1798: test_wait_for_health_ok: activate_osd td/ceph-helpers 1 2023-09-30T01:41:59.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2023-09-30T01:41:59.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-09-30T01:41:59.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-09-30T01:41:59.718 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/1 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-09-30T01:41:59.719 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:41:59.720 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:41:59.720 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:41:59.720 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:41:59.720 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:41:59.720 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:41:59.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:41:59.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:41:59.721 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:41:59.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:41:59.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:41:59.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:41:59.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:41:59.722 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:41:59.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-09-30T01:41:59.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-09-30T01:41:59.724 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/1 2023-09-30T01:41:59.725 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.1 2023-09-30T01:41:59.725 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-09-30T01:41:59.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:41:59.726 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/1/whoami 2023-09-30T01:41:59.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-09-30T01:41:59.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-09-30T01:41:59.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-09-30T01:41:59.726 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-09-30T01:41:59.752 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:59.750+0000 7faa155bf3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:41:59.769 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:59.766+0000 7faa155bf3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:41:59.771 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:41:59.766+0000 7faa155bf3c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:42:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-09-30T01:42:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:42:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-09-30T01:42:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:42:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:42:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:42:00.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:42:00.212 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:42:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:42:00.213 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:42:00.313 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:42:00.310+0000 7faa155bf3c0 -1 Falling back to public interface 2023-09-30T01:42:00.588 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:42:01.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:42:01.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:42:01.589 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:42:01.590 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:42:01.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:42:01.590 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:42:02.000 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:42:03.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:42:03.001 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:42:03.002 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:42:03.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:42:03.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:42:03.002 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:42:03.248 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:42:03.246+0000 7faa155bf3c0 -1 osd.1 14 log_to_monitors true 2023-09-30T01:42:03.440 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:42:04.442 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:42:04.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:42:04.443 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:42:04.444 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:42:04.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:42:04.444 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-09-30T01:42:04.855 INFO:tasks.workunit.client.0.smithi186.stdout:osd.1 up in weight 1 up_from 20 up_thru 0 down_at 16 last_clean_interval [10,14) [v2:127.0.0.1:6810/160573,v1:127.0.0.1:6811/160573] [v2:127.0.0.1:6812/160573,v1:127.0.0.1:6813/160573] exists,up 925f5ab8-5f7f-4498-8ee2-3c3aa6cdec69 2023-09-30T01:42:04.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:42:04.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:42:04.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:42:04.857 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1799: test_wait_for_health_ok: activate_osd td/ceph-helpers 2 2023-09-30T01:42:04.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2023-09-30T01:42:04.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-09-30T01:42:04.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-09-30T01:42:04.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-09-30T01:42:04.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/2 2023-09-30T01:42:04.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 ' 2023-09-30T01:42:04.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-09-30T01:42:04.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-09-30T01:42:04.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-09-30T01:42:04.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2023-09-30T01:42:04.858 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2023-09-30T01:42:04.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-09-30T01:42:04.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-09-30T01:42:04.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2023-09-30T01:42:04.859 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-09-30T01:42:04.859 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-09-30T01:42:04.859 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-09-30T01:42:04.861 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-09-30T01:42:04.861 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:42:04.861 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:42:04.861 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:42:04.861 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' 2023-09-30T01:42:04.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-09-30T01:42:04.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2023-09-30T01:42:04.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2023-09-30T01:42:04.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-09-30T01:42:04.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-09-30T01:42:04.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-09-30T01:42:04.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-09-30T01:42:04.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-09-30T01:42:04.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-09-30T01:42:04.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/2 2023-09-30T01:42:04.862 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-09-30T01:42:04.863 INFO:tasks.workunit.client.0.smithi186.stdout:start osd.2 2023-09-30T01:42:04.863 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --cluster ceph --fsid=779146aa-b0f9-4583-8ddd-7df6da70ad9d --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.76804/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-09-30T01:42:04.863 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/2/whoami 2023-09-30T01:42:04.864 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-09-30T01:42:04.864 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-09-30T01:42:04.864 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-09-30T01:42:04.864 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-09-30T01:42:04.891 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:42:04.886+0000 7f257dd363c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:42:04.918 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:42:04.914+0000 7f257dd363c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:42:04.919 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:42:04.918+0000 7f257dd363c0 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-30T01:42:05.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-09-30T01:42:05.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-09-30T01:42:05.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-09-30T01:42:05.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-09-30T01:42:05.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-09-30T01:42:05.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:42:05.329 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-09-30T01:42:05.330 INFO:tasks.workunit.client.0.smithi186.stdout:0 2023-09-30T01:42:05.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:42:05.330 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:42:05.493 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:42:05.490+0000 7f257dd363c0 -1 Falling back to public interface 2023-09-30T01:42:05.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:42:06.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:42:06.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:42:06.738 INFO:tasks.workunit.client.0.smithi186.stdout:1 2023-09-30T01:42:06.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-09-30T01:42:06.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:42:06.739 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:42:07.131 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:42:08.133 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:42:08.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:42:08.134 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-09-30T01:42:08.134 INFO:tasks.workunit.client.0.smithi186.stdout:2 2023-09-30T01:42:08.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:42:08.135 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:42:08.553 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-09-30T01:42:08.695 INFO:tasks.workunit.client.0.smithi186.stderr:2023-09-30T01:42:08.690+0000 7f257dd363c0 -1 osd.2 15 log_to_monitors true 2023-09-30T01:42:09.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-09-30T01:42:09.555 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-09-30T01:42:09.555 INFO:tasks.workunit.client.0.smithi186.stdout:3 2023-09-30T01:42:09.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-09-30T01:42:09.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-09-30T01:42:09.556 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-09-30T01:42:09.970 INFO:tasks.workunit.client.0.smithi186.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 16 last_clean_interval [0,0) [v2:127.0.0.1:6818/161156,v1:127.0.0.1:6819/161156] [v2:127.0.0.1:6820/161156,v1:127.0.0.1:6821/161156] exists,up a5386f0c-9e21-4393-abf6-d046a8a818f2 2023-09-30T01:42:09.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-09-30T01:42:09.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-09-30T01:42:09.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-09-30T01:42:09.971 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1800: test_wait_for_health_ok: wait_for_health_ok 2023-09-30T01:42:09.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1779: wait_for_health_ok: wait_for_health HEALTH_OK 2023-09-30T01:42:09.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1758: wait_for_health: local grepstr=HEALTH_OK 2023-09-30T01:42:09.972 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: delays=($(get_timeout_delays $TIMEOUT .1)) 2023-09-30T01:42:09.972 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: get_timeout_delays 300 .1 2023-09-30T01:42:09.972 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-09-30T01:42:09.973 INFO:tasks.workunit.client.0.smithi186.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-09-30T01:42:09.973 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-09-30T01:42:09.973 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-09-30T01:42:09.973 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-09-30T01:42:10.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1759: wait_for_health: local -a delays 2023-09-30T01:42:10.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: wait_for_health: local -i loop=0 2023-09-30T01:42:10.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:42:10.404 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:42:10.952 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 0 >= 27 )) 2023-09-30T01:42:10.953 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.1 2023-09-30T01:42:11.054 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:42:11.054 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:42:11.055 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:42:11.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 1 >= 27 )) 2023-09-30T01:42:11.596 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.2 2023-09-30T01:42:11.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:42:11.797 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:42:11.798 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:42:12.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 2 >= 27 )) 2023-09-30T01:42:12.354 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.4 2023-09-30T01:42:12.755 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:42:12.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:42:12.756 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:42:13.298 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 3 >= 27 )) 2023-09-30T01:42:13.298 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 0.8 2023-09-30T01:42:14.099 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:42:14.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:42:14.100 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:42:14.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 4 >= 27 )) 2023-09-30T01:42:14.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 1.6 2023-09-30T01:42:16.205 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:42:16.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:42:16.206 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:42:16.728 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 5 >= 27 )) 2023-09-30T01:42:16.729 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 3.2 2023-09-30T01:42:19.930 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:42:19.931 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:42:19.931 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:42:20.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 6 >= 27 )) 2023-09-30T01:42:20.457 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 6.4 2023-09-30T01:42:26.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:42:26.859 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:42:26.860 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:42:27.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 7 >= 27 )) 2023-09-30T01:42:27.385 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 12.8 2023-09-30T01:42:40.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:42:40.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:42:40.187 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:42:40.753 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 8 >= 27 )) 2023-09-30T01:42:40.753 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:42:55.754 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:42:55.755 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:42:55.755 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:42:56.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 9 >= 27 )) 2023-09-30T01:42:56.287 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:43:11.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:43:11.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:43:11.289 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:43:11.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 10 >= 27 )) 2023-09-30T01:43:11.810 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:43:26.811 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:43:26.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:43:26.812 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:43:27.340 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 11 >= 27 )) 2023-09-30T01:43:27.341 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:43:42.342 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:43:42.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:43:42.343 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:43:42.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 12 >= 27 )) 2023-09-30T01:43:42.893 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:43:57.895 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:43:57.896 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:43:57.896 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:43:58.431 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 13 >= 27 )) 2023-09-30T01:43:58.432 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:44:13.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:44:13.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:44:13.433 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:44:13.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 14 >= 27 )) 2023-09-30T01:44:13.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:44:28.962 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:44:28.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:44:28.963 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:44:29.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 15 >= 27 )) 2023-09-30T01:44:29.535 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:44:44.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:44:44.537 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:44:44.538 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:44:45.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 16 >= 27 )) 2023-09-30T01:44:45.073 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:45:00.075 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:45:00.075 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:45:00.075 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:45:00.601 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 17 >= 27 )) 2023-09-30T01:45:00.602 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:45:15.603 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:45:15.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:45:15.604 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:45:16.123 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 18 >= 27 )) 2023-09-30T01:45:16.124 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:45:31.125 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:45:31.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:45:31.126 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:45:31.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 19 >= 27 )) 2023-09-30T01:45:31.661 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:45:46.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:45:46.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:45:46.662 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:45:47.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 20 >= 27 )) 2023-09-30T01:45:47.177 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:46:02.179 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:46:02.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:46:02.180 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:46:02.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 21 >= 27 )) 2023-09-30T01:46:02.736 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:46:17.737 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:46:17.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:46:17.738 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:46:18.348 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 22 >= 27 )) 2023-09-30T01:46:18.349 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:46:33.350 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:46:33.351 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:46:33.351 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:46:33.878 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 23 >= 27 )) 2023-09-30T01:46:33.878 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:46:48.880 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:46:48.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:46:48.881 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:46:49.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 24 >= 27 )) 2023-09-30T01:46:49.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:47:04.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:47:04.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:47:04.409 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:47:04.957 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 25 >= 27 )) 2023-09-30T01:47:04.958 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 15 2023-09-30T01:47:19.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:47:19.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:47:19.960 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:47:20.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 26 >= 27 )) 2023-09-30T01:47:20.483 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_health: sleep 4.5 2023-09-30T01:47:24.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: wait_for_health: loop+=1 2023-09-30T01:47:24.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: ceph health detail 2023-09-30T01:47:24.985 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_health: grep HEALTH_OK 2023-09-30T01:47:25.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_health: (( 27 >= 27 )) 2023-09-30T01:47:25.530 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1764: wait_for_health: ceph health detail 2023-09-30T01:47:26.080 INFO:tasks.workunit.client.0.smithi186.stdout:HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:47:26.081 INFO:tasks.workunit.client.0.smithi186.stdout:[ERR] MGR_MODULE_ERROR: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:47:26.081 INFO:tasks.workunit.client.0.smithi186.stdout: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:47:26.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1765: wait_for_health: return 1 2023-09-30T01:47:26.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1779: wait_for_health_ok: return 1 2023-09-30T01:47:26.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1800: test_wait_for_health_ok: return 1 2023-09-30T01:47:26.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: run_tests: teardown td/ceph-helpers 1 2023-09-30T01:47:26.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2023-09-30T01:47:26.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2023-09-30T01:47:26.093 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2023-09-30T01:47:26.094 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-09-30T01:47:26.094 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-09-30T01:47:26.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-09-30T01:47:26.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-09-30T01:47:26.094 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-09-30T01:47:26.210 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-09-30T01:47:26.210 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-09-30T01:47:26.212 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-09-30T01:47:26.212 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-09-30T01:47:26.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-09-30T01:47:26.214 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-09-30T01:47:26.215 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-09-30T01:47:26.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:47:26.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-09-30T01:47:26.217 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-09-30T01:47:26.218 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:47:26.219 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:47:26.221 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2023-09-30T01:47:26.221 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2023-09-30T01:47:26.221 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2023-09-30T01:47:26.222 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/ceph-helpers/mgr.x.log td/ceph-helpers/mon.a.log td/ceph-helpers/osd.0.log td/ceph-helpers/osd.1.log td/ceph-helpers/osd.2.log /home/ubuntu/cephtest/archive/log 2023-09-30T01:47:26.223 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2023-09-30T01:47:26.235 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-09-30T01:47:26.235 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-09-30T01:47:26.235 INFO:tasks.workunit.client.0.smithi186.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76804 2023-09-30T01:47:26.235 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76804 2023-09-30T01:47:26.236 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-09-30T01:47:26.236 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-09-30T01:47:26.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: run_tests: return 1 2023-09-30T01:47:26.237 INFO:tasks.workunit.client.0.smithi186.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: : exit 1 2023-09-30T01:47:26.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:47:26.238 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2023-09-30T01:47:26.238 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-09-30T01:47:26.488 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi186 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=93dce5e17918f8aca65993a7a2dc40233a47b481 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2023-09-30T01:47:26.560 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=eeea91cfe2064ed89321d1cb7466cb92 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on smithi186 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=93dce5e17918f8aca65993a7a2dc40233a47b481 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2023-09-30T01:47:26.563 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T01:47:26.573 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T01:47:26.573 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T01:47:26.620 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T01:47:26.628 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T01:47:26.629 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:47:26.656 INFO:teuthology.orchestra.run.smithi186.stdout: remote refid st t when poll reach delay offset jitter 2023-09-30T01:47:26.656 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================== 2023-09-30T01:47:26.656 INFO:teuthology.orchestra.run.smithi186.stdout:*hv01.front.sepi 44.4.53.2 2 u 61 64 377 0.108 1.021 0.115 2023-09-30T01:47:26.656 INFO:teuthology.orchestra.run.smithi186.stdout:+hv02.front.sepi 63.231.80.2 3 u 7 64 377 0.092 1.053 0.120 2023-09-30T01:47:26.657 INFO:teuthology.orchestra.run.smithi186.stdout:+hv03.front.sepi 74.6.168.72 3 u 7 64 377 0.082 0.275 0.079 2023-09-30T01:47:26.657 INFO:teuthology.orchestra.run.smithi186.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-09-30T01:47:26.657 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T01:47:26.666 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T01:47:26.666 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T01:47:26.674 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T01:47:26.683 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T01:47:26.692 INFO:teuthology.task.internal:Duration was 3429.589407 seconds 2023-09-30T01:47:26.692 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T01:47:26.700 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T01:47:26.701 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T01:47:26.753 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T01:47:26.753 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2023-09-30T01:47:26.753 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T01:47:26.765 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T01:47:26.765 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T01:47:26.813 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T01:47:26.814 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T01:47:26.871 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T01:47:26.880 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T01:47:26.880 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T01:47:26.893 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T01:47:26.902 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:47:26.949 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2023-09-30T01:47:26.979 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T01:47:26.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:47:26.985 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T01:47:26.994 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T01:47:26.996 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406800/remote/smithi186 2023-09-30T01:47:26.997 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T01:47:28.059 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T01:47:28.060 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T01:47:28.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T01:47:28.110 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T01:47:28.111 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T01:47:28.120 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T01:47:28.120 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T01:47:28.128 INFO:teuthology.orchestra.run.smithi186.stdout: 924112 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 30 01:47 /home/ubuntu/cephtest 2023-09-30T01:47:28.128 INFO:teuthology.orchestra.run.smithi186.stdout: 1053753 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 30 00:57 /home/ubuntu/cephtest/mnt.0 2023-09-30T01:47:28.129 INFO:teuthology.orchestra.run.smithi186.stdout: 1065548 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 30 01:17 /home/ubuntu/cephtest/mnt.0/client.0 2023-09-30T01:47:28.129 INFO:teuthology.orchestra.run.smithi186.stdout: 1194915 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 30 01:17 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-30T01:47:28.129 INFO:teuthology.orchestra.run.smithi186.stdout: 1194916 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Sep 30 01:47 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2023-09-30T01:47:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:47:28.131 INFO:teuthology.orchestra.run.smithi186.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-09-30T01:47:28.131 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi186 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-09-30T01:47:28.136 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T01:47:28.147 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T01:47:28.157 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi186 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-09-30T01:47:28.158 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T01:47:28.185 DEBUG:teuthology.nuke:shortname: smithi186 2023-09-30T01:47:28.186 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T01:47:28.208 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406800', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:41:07.886654', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-09-30T01:47:28.252 INFO:teuthology.orchestra.console.smithi186:Power off 2023-09-30T01:47:28.253 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T01:47:28.275 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2023-09-30T01:47:28.276 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:47:28.299 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2023-09-30T01:47:32.301 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:47:32.413 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2023-09-30T01:47:36.417 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T01:47:36.526 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2023-09-30T01:47:36.627 INFO:teuthology.orchestra.console.smithi186:Power off completed 2023-09-30T01:47:36.733 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} duration: 3429.5894067287445 failure_reason: 'Command failed (workunit test misc/test-ceph-helpers.sh) on smithi186 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=93dce5e17918f8aca65993a7a2dc40233a47b481 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=eeea91cfe2064ed89321d1cb7466cb92 status: fail success: false 2023-09-30T01:47:36.733 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T01:47:36.831 INFO:teuthology.run:FAIL